9 posts tagged with "changelog"

Cross-instance Padif analyses​

The major novelty with this release is the possibility to perform Padif analyses which include not only documents from your own instance but also from other ProFID instances.

This feature will allow users to explore possibly related cases in other countries, empowering them to further their investigations by reaching out to their colleagues.

Disclosed data​

As a preamble, it is already worth mentioning that the feature only discloses the strict minimum of metadata about documents and series to users of other instances:

• The document's sequential identifier, containing the document's creation year and a sequential number.
• The identifier and name of the series any of the documents belong to.
• The name and email of document creators and series managers. Those are disclosed so that users can contact the right person in case a document or a series is of particular interest.

In particular, users don't have access to any images related to documents from other instances. Any computation required to calculate the similarity between documents is done privately on the server.

Performing a cross-instance analysis​

To start a cross-instance analysis, go to an existing Padif analysis and select the "External instances" action from the dropdown menu.

Then select from the list of available instances the one you want to include in the analysis. The number in parentheses shows you the number of documents that match the document set filters of the analysis in the other instance.

Once saved, it will take about 30 minutes or more until you see the result of the cross-instance analysis, depending on when the next run is scheduled, how many other analyses are in queue and how many documents and regions of interest the analysis has to be performed on.

For each external instance selected, you will see a new tab containing the usual Padif dendrogram, but with documents from the other instance listed, as well as their associated series.

Opting out​

By default, all instances except for ProFID series (which does not hold documents, only series), can be included in cross-instance Padif analyses.

It is, however, possible to opt out of sharing by emailing us at support+profid@zakodium.com.

Opting out will prevent both other instances from using your data and yourself to use the data of any other instance.

Other Padif improvements​

New ROI edition interface​

The page to edit regions of interest (ROIs) for Padif analyses has been completely redesigned.

You can now work on all the ROIs at once instead of removing and adding them one by one. It works similarly to the page for editing anonymized regions when importing scans.

You can also choose the color of the ROIs, which was not possible before.

Status information​

We added context on the state of a Padif analysis, so that you are aware when things are missing or in an errored state in the report:

• The number of documents included in the report.
• The time at which the last execution happened, or if the analysis is being currently executed.
• The list of documents which are part of the analysis but aren't in the report, along with the reason why.
• The number of regions of interest which are configured in the analysis but are not yet part of the report.

Performance improvements​

The Padif analyses are computationally expensive, and it can take some time to generate the report.

We have audited the performance bottlenecks and made some improvements to fasten the process.

Security improvements​

More minor security concerns have been fixed based on the second security audit. Notably, users are now required to set a password with at least 10 characters, and users can see feedback on the strength of the password they chose.

Bug fixes and minor improvements​

Minor improvements​

• The activation / deactivation of a Padif analysis now has to be confirmed to make the operation safer.
• We facilitated bug reports by providing a direct mailto link in error alerts.
• For admins, we added the ability to filter users by role.
• The scan import process was made more robust, handling the presence of a white crease at the top of the canvas.
• In the document suggestions tab, if the document's serial number matches with others in the database, provide basic information about those documents and a link to them.

Bug fixes​

• We fixed an error which prevented labels from being renamed.
• We prevented ROIs which exist on the Padif analyses but are not in the report, to be selected in the report.
• We prevented the deletion of a document if it is a reference document of a Padif analysis.
• When viewing a document, the suggestion tab lists the Padif analyses in which the document is involved. We fixed a bug which made the list empty even when there were linked Padif analyses.

Sharing series​

It is now possible to share series with other ProFID instances, by securely exporting them and letting the recipient import them into their own instance.

Exporting a series​

When sharing a series, you will be invited to enter a password to protect the file, and to choose which data you want to include in the export:

• Series images
• Series attachments
• Last published alert

General information like the name of the series, as well as the series material and contextual profiles, are always included in the export.

Importing a series​

With a .pfd file in hand, the recipient can go to "Series > Import series" in the top menu, and upload the file.

Provided that the password is known to the recipient, they will be able to import the series into their instance.

The new identifiers created for the imported series, are generated in the same way as regular series. They will contain the instance code of the instance in which the series is imported, the year and an incremental number. However when going to the series' page, you will notice that they will contain a reference to the exported series.

Imported series have the same permission rules as any other series, and they can be edited afterwards or have documents linked to them.

Series import / export is the first step towards a more open and interconnected ProFID. We hope that it will foster collaboration between countries and allow for more links to be established, helping the fight against organized crime.

New Padif​

The new Padif, the automated document comparison algorithm to help ProFID users attribute documents to series, is now out of experimental phase and generally available to all users. The old version of Padif has been removed.

Padif rewrite​

Padif, originally written in Python by Sydney Barthe, who was a student at Ecole des sciences criminelles, University of Lausanne, did not evolve since it was first integrated into ProFID, since the technology stack is very different from the rest of ProFID. Some known bugs were never fixed.

The new Padif, entirely rewritten in JavaScript, is more tightly integrated with the rest of the codebase and will give greater autonomy to the development team to fix bugs and evolve the algorithm. It improves some key parts of the process. For example the algorithm for aligning regions of interest in between documents has been rethought and improved.

New Padif features​

The new Padif allows for regions of interest to be included / excluded from the results, without the need to re-run the comparison. This will make the process of finding the right regions of interest for the analysis less tedious. It also allows to choose between different distance aggregation algorithms for creating the dendrogram.

Security improvements​

A second security audit was conducted on the ProFID software.

Various minor security concerns have been identified and fixed to make ProFID even more secure.

Bug fixes​

• Fix incoherent filter indicators in scan list page.
• Improve guiding borders around document QR codes to ensure they are cut correctly.
• Make sure only admin or editors have access to the list of ProFID users.

Notable changes​

New workflow to import scans​

This workflow is split into 4 steps:

1. Extraction
   Finds the borders of the document and extrapolates the best coordinates and angle to crop the document.
2. Rotation
   Using a machine learning model, re-orients the document so that it is upright.
3. MRZ extraction
   Using machine learning models, finds the MRZ and reads its characters.
4. Anonymization
   Using machine learning models, finds and prepares the anonymization of faces, signature, barcodes and MRZ text on the document.

Each step can be manually corrected if needed.

If an MRZ is found, the user must validate that it was correctly read.

QoL improvements​

• Document sets can now be deleted if they are not associated with a Padif analysis.

• Display creator and editor details (date, time, mail link) on Scan, Document set, Label, Padif analysis, and Series.

• The web page title (in browser tab) is updated with your navigation. Now it's easier to differentiate multiple ProFID opened tabs.

  

• Improved the scan details page.

• Label's descriptions are now optional.

Bug fixes​

General​

• Fixed a bug that prevented the edition of documents from Kosovo. The Kosovo country code is now correctly set to XKK.
• When an unexpected error happens, pages with tabular data will now show the error instead of the previous data.

Document​

• In document standardized image edition page, ensure text in buttons do not wrap to improve readability.
• Labels filter displays correct associated documents count.

Admin​

• In users management pages, replace wrong links in breadcrumb from /users to /admin/users.

Notable changes​

Navigation breadcrumb on almost every page of the application​

The page layout has also been improved and made more consistent.

Breadcrumb segments link to the parent page.

Improve ROI creation in Padif analysis​

The ROI creation module has been completely revamped. It now includes a preview of existing ROIs and a usage guide.

Pages with tabular data come with search, sort and filters​

The ability to search, sort, and filter has been added to the following tables:

• Scans
• Padif analyses
• Users
• Services
• Seizure authorities

Security fixes​

ProFID has been audited by an independent security company. This release includes various fixes and best practices to improve the security of the application.

One of these changes might impact users: the allowed file types for uploads are now restricted.

• For scan uploads and document extra images: PNG, TIFF, and JPEG.
• For series attachments: PNG, JPEG, BMP, PDF, and TXT.
• For series alert image uploads: PNG and JPEG.

Other small improvements​

If you encounter any unexpected error in the application, it will show you a unique request identifier. Please include this identifier if you make a support request.

Manual and dynamic document sets​

Manual document sets have long existed in ProFID, but lacked some key features and were not very easy to construct. This release significantly improves the usability of setting up document sets, and introduces dynamic document sets, a new type of document set which automatically updates the list of documents based on user-chosen filters. This opens up new use cases for Padif analyses, where new documents can be compared to other documents in the set as soon as they are added.

List of document sets​

The list of document sets is now presented in a table, with search, sort and filter features.

Manual document sets​

Manual document sets used to be hard to construct because:

• The metadata of documents was not accessible from the document set interface.
• The list of documents was not searchable or sortable.
• It was not possible to exclude documents, which made them appear among valid candidates.

All those issues have been addressed in this release. Document set members are now displayed in a table with metadata, candidates can be searched or sorted, and documents can be excluded from the candidate list. Candidates are managed in a large slide-over panel. On reasonably large screens, this allows to see the documents already in the set and candidates at the same time.

Excluded documents can still be re-included to the candidate list if a mistake was made.

Dynamic document sets​

Dynamic document sets is a brand-new feature which allows users to have document sets populated without having to manually add each document to the set. The principle is simple, you select a set of filters similarly to how you would search for documents, and the set will automatically be populated with documents matching the filters.

Similarly to manual sets, it is possible to exclude documents from the set. Doing so will prevent the document from being part of the set, even though it matches the filters.

As mentioned in the previous release, there are valid use cases for using labels as filters for dynamic document sets, and this is now possible.

Documents of a dynamic set are displayed in a similar manner than manual sets.

The filters of a document set can still be edited after the set has been created.

Other document set improvements​

We also added a couple of other improvements to document sets:

• Document sets can be renamed
• A document can be added to a manual set directly from the document's page.

Padif​

The selection of the reference document has been improved. The images are no longer obstructed by labels and additional information about the document is displayed.

Documents​

• It is now possible to assign / remove labels to / from multiple documents in one action from the document list page.
• The search input in document list page is now split into terms which can be looked up in all relevant fields. The result contains the items which matched all terms.

Series​

As for the document list page, the search input in the series list page list is now split into terms which can be looked up in all relevant fields. The result contains the items which matched all terms.

Alert​

New toolbar button to select predefined sentences in "Measures" field.

Other small improvements​

• Display a warning when the user attempts to change the scope of a label.
• Invalidate user session when an admin resets the user's password.
• Padif analyses can now be deleted.

Bug fixes​

• Add and fix translations on series alerts.
• Fixed a crash in the alert edit page which occurred in some circumstances when the edited language was changed.
• Fixed an error in the alert page on rapid identification image edition modal form which occurred when the form was submitted without setting the image.
• Publishing an alert now changes the edition date of the series.
• Fixed invalid values on home stats page when no documents were assigned to any series.
• Allow removing service from users in the admin list of user page.
• Fixed the behaviour when clicking on label. The list of documents is now filtered on this label as expected.
• Fixed missing count for the unspecified option in filters.
• Fixed the download button on the scan details page. It was downloading a lighter version instead of the original one.

ProFID series alerts​

This is the biggest addition to this release. There is a dedicated article on series alerts.

Document labels​

A label is an arbitrary keyword that can be associated to any number of documents. This is useful for further characterizing documents with attributes which do not exist as fields in the document. One key advantage of labels is that they are searchable from the document list page.

Before you start creating labels, it is important to concert with other users of your ProFID instance to agree on a common set of labels. This will ensure that labels are used consistently and that they are useful to everyone.

Labels can be used for a variety of purposes. For example, you can use them to organize work: a non-expert might add the "requires analysis" label to documents which need input from a document analysis expert. The label can then be used by experts to filter the document list page to only show documents which require analysis.

Furthermore, labels can be scoped. Scoped labels are special labels which have a constraint on the document type, country, and/or fraud type on which it can be applied. One potential application of scoped labels is to categorize documents based on their comparability in terms of automated image processing. For example, one might create a scoped label that only applies to Swiss IDs and differentiates the "2023 version" of the document from the older ones which have a radically different design. In future versions of ProFID, categorising documents this way will enable the use of "smart" document sets, which will automatically update based on those labels or other filtering criteria. The update of document sets will automatically trigger Padif analyses to be re-run.

Rich text editor​

Until today, it was (almost) not possible to format text in ProFID. This is now officially possible thanks to the new rich text editor. It allows a wide range of possibilities to format text: headers, text alignment and colours, bullet points, etc. It is available in the following places:

• The different alert fields: general information, rapid identification element and measures.
• The series material and contextual profile fields.

The previous version of ProFID allowed to use a syntax called "markdown" to format text in the series material and contextual profile fields. This was a non-documented feature but was used by some users. Those fields have been migrated to the new rich text editor for seamless transition.

New dedicated pages for images and attachments​

ProFID now features a new dedicated edition interface for standardized and extra images on documents, as well as images and attachments on series. This new user-friendly interface creates a clearer delimitation between view and edit pages.

Prevent lost work​

ProFID has forms to fill data about documents and series. It is not uncommon to spend time filling a form only to accidentally lose everything by navigating to another page or by closing the browser's tab. ProFID will now warn you if you are about to lose your changes.

Improved session restoration​

For security, ProFID logs you out after 30 minutes of inactivity. An overlay inviting you to log in again appears on the page when that happens. When multiple tabs are open, you had to log in again in each tab. This is no longer the case: you will only have to log in once and all tabs will be automatically restored!

Improved document and series lists​

We show more fields in the document list (document number, operation) and on series (aliases). Before this addition, some fields were searchable but not displayed in the results, creating confusion.

New type of light for extra images​

We added 2 new types of light for extra images, in alignment with the FADO terminology:

• Coaxial
• Oblique

Other new pages​

• We added a new scan details page available from the scan list.
• We added a standardized image details page available from a document detail's page. Hover the image to access the link.

Bug fixes​

• Fixed a bug which allowed multiple users with the same email address to be created, if they were written with different casing.
• Fixed erroneously updating document's date of addition to a series when non-series fields are updated.

Sequential ids for documents and series​

From now on, sequential identifiers are automatically assigned to documents and series. They contain the character "D" or "S", the year of creation and a number that starts at 1 and is reset every year.

Additionally, the identifier for series contains a three-letter code for the ProFID instance that created it (usually a country code). This allows to share series identifiers between ProFID instances with no ambiguity.

• Example of a document identifier: D-23-0001
• Example of a series identifier: S-DEU-23-001

Other changes​

• Series are no longer created while describing a document. If a document is associated to a series that doesn't exist yet, the series will be created when the document is saved.
• In the series list, the "Updated at" column no longer reflects all changes to a series. This field is now only updated when the series' list of documents is modified or when its contextual or material profile are updated.
• In documents and series search filters, clicking the "Reset" button now also resets the text search term.
• Various bug fixes and improvements to the user interface.

In this release, we have added the possibility to rename series and made a few improvements to the user interface.

Renaming series​

It is now possible to change the name of a series:

When a series is renamed, an alias is automatically created with the old name. It can be removed afterwards by editing the series.

A series can have multiple aliases. They are displayed in the series's page and search boxes now also search within aliases in addition to the series names.

User interface improvements​

• For both series and documents, the main actions you can do are now all located on the top-right of the page:

  

• We also changed the way standardized document images can be rotated or deleted. The buttons now appear when going over the image with the mouse.

Bug fixes​

• Fixed a bug which was preventing to submit forms with a validation error. It is no longer necessary to refresh the page to be able to submit the form again.

12 August 2022​

General​

• Many terms have been updated to follow the FADO terminology.
• Visual redesign of the entire user interface.
• Reduced the validity time of a user session to 30 minutes.
• Updated the user guide to version 3.3.
• Bug fixes and various improvements.

Documents​

• Added the possibility to attach extra images (not acquired by scan) to a document.
• Added "Unknown" option in the "Type of fraud" field.
• Added the filters "Scan availability" and "Extra image availability" in the document search.

Series​

• Added the possibility to merge series.
• Added the possibility to permanently delete a series.

Administration​

• Added the possibility to delete a user who has never activated their account.
• Added filters and the ability to sort the user list.

15 March 2021​

• The interface is now translated into Dutch.

03 March 2021​

• Added the possibility to delete individual images from documents. This operation is irreversible and leads to the deletion of the associated scan.
• Added the possibility to delete scans which failed to be imported (ERROR status).
• Added the possibility to directly access the following pages with their own URL: results of Padif analyses, front and back of a document, suggestions for a document.
• Document filtering options are now kept in the URL, allowing to share or bookmark a particular search.
• Added more filtering options for documents (technical analysis fields).
• Added links to the homepage allowing to filter documents by service, document country or presumed nationality of document holder.
• Added a button (magnifying glass) on the document description page allowing to do a filtered search using the values of the technical analysis.
• Added links on the document description page pointing to active Padif analyses that the document belongs to.
• Added the possibility to rotate document images.
• Improved display of the result for Padif analyses.
• Added a "Padif distances" section in the suggestions page for a document. If the document belongs to at least on Padif analysis, the closest (smallest distance) documents to it will be displayed.

18 November 2020​

• The new scan canvas can now be used to import documents. The old canvas is still supported and the format will be detected automatically by the system. The extraction of reference colors from the old canvas was improved.
• Added a help page with videos documenting document sets and Padif analyses.
• Fixed a bug that prevented to correctly save the "Unspecified" option on MRZ, Background and Bitmap code document fields. This option will now also be selected by default for all new documents.
• Display the date and the name of the user who created a series or a document in their respective pages.
• Administrators now have the possibility to create new services and seizure authorities.
• Administrators can now delete documents and all associated images and scans.