Searching years of municipal records without moving a file
When an organization decides its documents are too hard to find, the usual proposal is a migration: buy a document management system, move everything into it, and file new documents there from now on. We took a different route with Sasly Knowledge, and this article explains why.
The trouble with migration
A migration asks every department to change how it works on the same day. Staff have years of habits tied to folder paths. Other systems, spreadsheets and shortcuts point at those paths. Moving the files breaks all of them at once.
It also takes a long time, and during that time nobody is finding documents any faster. In practice many migrations stop partway, and the organization ends up with two places to look instead of one.
Indexing in place
The alternative is to leave every file where it is and build a search index beside it. The server is given read-only access to the shared folders. It opens each file, extracts the text, reads scans with text recognition, and records what it found. The files are not moved, renamed or changed.
Staff then search from a browser. A result shows the document and its full folder path, so people keep working with the record in its original location. Each night the server checks for files that were added, changed or removed.
Three things that have to be right
Permissions
A search index that ignores permissions is a leak. If a clerk can find the title of a document they are not allowed to open, something confidential has already been disclosed. In Sasly Knowledge the filter is applied inside the search itself, so a file a person may not see does not appear in their results, their counts or their suggestions.
Honesty about what was skipped
Every collection has files that cannot be read: a corrupted PDF, a password-protected spreadsheet, a photograph with no text. A search tool that silently skips them gives people false confidence. The administration screen lists how many files are searchable, how many could not be read, and the reason for each.
Scale on ordinary hardware
A municipality does not want to buy a rack of servers to search its own documents. We tested Sasly Knowledge with a generated library of 100,000 files on a laptop with 16 GB of memory. Searches came back in 13 to 33 milliseconds. Real collections vary, and we expect to tune for each one, but the starting point does not need special equipment.
Where this helps most
Access-to-information requests are the clearest case. Under Ontario’s municipal freedom of information legislation, an institution generally has 30 days to respond to a request. Much of that time can go to locating the responsive records in the first place. When locating them is a search that takes seconds, the days go to review and decisions, which is where they belong.
The same applies to a new employee trying to learn what the department decided five years ago, and to a manager who wants to know what a shared drive holds before it is cleaned up.