InterOp Caching Implementation

System Overview

• system starts, load all metadata to cache, load all metadataurlByQuerystring to cache
• Vector resultsBySql = smanager.getMatchedMetadataUrlListFromDB(smanager.queryToSQL(req), db)
  - local db sql search get matched metadata URL;
• Vector v = smanager.getMetadataLinksByQuery((dinfo.getName()+":"+helper.htToString(ht)).toUpperCase());
  if (v != null && !v.isEmpty()) resultsByQuerystr.addAll(v);
  - check local cache based on querystring, if not goto remote DL populate results (seems still based on old
  way search, if results empty refresh, untill get results then also populate resultsByQuerystr, searchAgent will do the update metadatalinks based on querystring?)
• display either merged, complete, or original view
• upon request or every hour, save metadata in cache

Caching Enhancement

• metadatalinksbyquery if in cache, don't go to remote dl, need to store in db? if so, not fresh anymore?
• add two more fields for cache replacement algorithm
  alter table dc add (date_last_used date, total_usage number(5))
• select allmetadatafromdb use cache sizelimit (order by total_usage desc, date_last_used desc)
• change procedure dc_AddNew to update those two fields too
• cache replacement algorithm:
  • initial parameter: cache size, cache keep safe size
  • when first start: load from db order by date_last_used, total_usage and pick only cache size
    • String orderBy = " ORDER BY total_usage desc, date_last_used desc";
    • String selectMetadata = "SELECT internalID,identifier,archive,id_within_archive,datestamp,title," +
      "creator,creator_affiliation,subject,description,publisher,publication,keyword,category," +
      "contributor,type,format,source,language,datelastupdated,additional_fields,status, " +
      "date_last_used, total_usage FROM dc ";
  • each time when user view a metadata, update date_last_used and total_usage
  • if cache full, remove least used from cache and save to db(first sort by date,keep safe,then sort by usage), or use weight, combine date and usage?
  • cache size and keep safe size can changed at runtime

Demo Steps

1. go to /interop/servlet/dlibServlet?formname=info page verify cache size and keep safe size, and check metadata in cache /dlibServlet?formname=showallmetadata
2. change cache size to 10 /dlibServlet?maxmetadataincache=10, cache keep safe size to 8 (so remove least used one from the two oldest ones) /dlibServlet?metadataincachekeepsafe=8, and verify metadata in cache /dlibServlet?formname=showallmetadata
3. search CogPrints limit results 5, verify metadata in cache
4. search NEEDS limit results 5, verify metadata in cache
5. change cache keep safe size to 9, search CogPrints limit 1, now should always remove oldest from cache