Reworking the data collector

The iteration of the GeoJSON data collector in the previous chapter was a straightforward capture of data to the database. In order to make the data easier to handle for other applications, the program will be improved to make a table of individual features listed.

Adding a new data table

The separated features in the database will be stored in a new table named, dm_quakefeatures, as follows:

CREATE TABLE dm_quakefeatures
(
  qufeat_id text NOT NULL,
  geojson text NOT NULL,
  modified_date timestamp without time zone DEFAULT now(),
  CONSTRAINT dm_qufeat_pkey PRIMARY KEY (qufeat_id)
)
WITH (
  OIDS=FALSE
);
ALTER TABLE dm_quakefeatures
  OWNER TO webquakeuser;

The SQL statements for creating the database are found in the SQL folder of the sample code.

Filtering the data

The DataMonitor class in the file quakemonitordb.dart has an updated fetchData method, which individually stores the underlying features after storing the raw JSON to the database:

GeoUpdate update = new GeoUpdate(JSON.encode(dataset));

log.fine("Features to process ${update.features.length}");

for (int i = 0; i < update.features.length; i++) {
  GeoFeature feature = update.features[i];

  await daoGeoJson.storeFeature(feature.id, feature.toJson());
}

The GeoUpdate class, which is found in geoupdate.dart, takes a JSON string as a parameter in the constructor. The following excerpt from the constructor shows how it extracts and adds the individual features to the collection:

GeoUpdate(String json) {
  src = json;
  features = [];

  if (src.length > 0) {
    try {
      Map rawJsonObj = JSON.decode(src);

      if (rawJsonObj["features"] != null) {
        var items = rawJsonObj["features"];

        items.forEach((Map i) {
          features.add(new GeoFeature(i));
        });
      }
    } catch (exception) {
      print("Error decoding JSON.");
      print("$src");
      print(exception);
    }
  }
}

The GeoFeature class, which is found in the geofeature.dart constructor, takes a Map object that is populated with the properties of the feature. The purpose of this is to extract the required information about the feature that needs to be stored for our application, as shown in the following code snippet:

GeoFeature(Map newProperties) {
  properties = newProperties["properties"];
  geometry = newProperties["geometry"];
  id = newProperties["id"];
  time = properties["time"].toString();
  title = properties["title"].toString();
  type = properties["type"].toString();
  mag = properties["mag"].toString();
  place = properties["place"].toString();

  url = properties["url"].toString();
  detail = properties["detail"].toString();
}

As the preceding excerpt shows, this is not a complicated object that arranges data into convenient fields.

Converting the feature to JSON

To convert the object back to JSON, a specific method will be added, called toJson, as shown in the following code snippet:

String toJson() {
  Map out = {};
  out["properties"] = properties;
  out["geometry"] = geometry;
  return JSON.encode(out);
}

The key fields of properties and geometry are added to Map before being passed to the encoding function and returned to the caller.

Improving the data maintenance

Since there is a new table in the database, we will want to update the maintenance program maintenance.dart to clear both tables, as shown in the following code snippet:

deleteRecords() async {
  var dbConn;
  try {
    dbConn = await connect(uri);

    var query = "delete from dm_quakefeed";
    await dbConn.execute(query);

    query = "delete from dm_quakefeatures";
    await dbConn.execute(query);
  } catch (exception, stacktrace) {
    log.severe("Exception getting info.", exception, stacktrace);
    print(exception);
  } finally {
    dbConn.close();
  }
}

This is carried out by using another delete SQL statement, and reuses the existing connection to the database.

Storing the single feature

The storage of the feature will take two separate queries. The first will use the ID field to check if it is currently in the table. If it is, then the function will exit without storing the JSON string. Let's have a look at the following code snippet:

 storeFeature(String featureID, String json) async {
  var dbConn;
  try {
    dbConn = await connect(uri);

    var query = """select count(*) as Count
     from dm_quakefeatures where qufeat_id ='$featureID'
    """;
    var results = await dbConn.query(query).toList();

    if (results[0][0] != 0) return;

    await dbConn.execute(
        'insert into dm_quakefeatures (qufeat_id, geojson) values (@qufeat_id, @geojson)',
        {'qufeat_id': featureID, 'geojson': json});
  } catch (exception, stacktrace) {
    log.severe("Exception storing Feature.", exception, stacktrace);
    print(exception);
    print(stacktrace);
  } finally {
    dbConn.close();
  }
}

The first query returns the number of records that match the ID. The result returned is converted to a list format, and the value is accessed via [0][0]—the first record and first field. If the result is not zero, then the method returns without further action.

The second SQL operation stores the feature in the database by using an insert statement.

Running the data application

The application can be run as usual from the command line. You may wish to run the data collector like this in order to save having multiple IDEs open for a long period of time. Let's have a look at the following code snippet:

$ cd QuakeMonitorDB/
QuakeMonitorDB$ dart bin/main.dart
Starting QuakeMonitor DB version...

This shows the launching of the QuakeMonitor program in a terminal. If a close look is taken at the processes running on the system, the resources used by the Dart VM can be seen, as shown in the following screenshot:

The Dart process launched for the QuakeMonitor measures approximately a mere 64 megabytes, and starts nearly instantly.