All the things you need to know
(2015-04-30) Contributors are developers who helped the project by giving their time and energy to code, debug, test, and generally improve the WXReplication project.
Sponsors are people or companies who gave money to help the contributors eat from time to time :-).
If you are interested in becoming one or the other, please contact me at email@example.com or via any of the other means of contact available on my contact page.
- NOBODY YET.... Don't be shy, you know you want your name to be here!!!
- Fabrice Harari, DataTale LLC, Guadeloupe, FWI: everything else... You can contact me at firstname.lastname@example.org about WXReplication or anything else..
(WXEDM is itself based on WXReplication, another open source project based here. However, Sponsors and Contributors of WXReplication are of course not listed here but in the WXReplication chapter of this site)
Sponsors et contributors (...read more...)
As many of you are aware, I presented a course on WXReplication during WXDevCon 2014. Tim once again took the matter in is hands recorded the whole thing, using a mixture of TeamViewer, Snaggit, and plain sound hardware.
Anyway, together, these 5 videos will take nearly 8 hours of your life that you'll never get back...
So watch them at your own peril (or if you really want to know more about WXEDM).
Oh! Did I mention they are free? ....Enjoy...
Part 1 - Part 2 - Part 3 - Part 4 - Part 5
WXDevCon 2015 presentation videos (...read more...)
WXEDM: What is that?
- It's an Open Source Electronic Document Management software,
- it's written in WinDev (mostly by me)
- it's based on WXReplication, which means that it is able to use replication to disseminate and backup the documents
- it's the subject of my main presentation at the next WXDevCon (2015)
- As I write this article (April 2015), it is in its infancy... This means that no source download is available yet and no source download will be available for the next few months... If you are in a hurry to see it come to live, please contact me for sponsoring opportunities...
What is it? (...read more...)
(2015-0927 In this article, I am going to list a ton of features that I would like to include in the project... They will or will not be implemented depending on available time and sponsoring. The only thing that is sure about this project is that a working version of WXEDM will be available for WXDevCon 2015 (Autumn 2015) and that I will be presenting it during the conference...
DONE means that a basic version of this functionality has been coded and is functional.
TEST: means that a proof of concept of the functionality has been coded and needs to be integrated in WXEDM and fleshed out.
TODO: the whole thing needs to be developed. This DOES NOT mean that this function will make the cut for WXDevCon or even that it will EVER be developed... Once again, if you need something specific at a specific date, please contact me directly for sponsoring opportunities.
DONE: Basic Analysis: replication files + documents, users, blobs, help, etc.
DONE: Basic style design/choice: based on a modified neo light template
DONE: Basic program behavior design: touch ready UI, with non modal windows hosting sets of internal windows. This design should be easily usable both in traditional and touch systems. Migration of functions to mobile setting should be much easier this way.
DONE: Internal help system allowing to create and update the help system and view it from inside the program (multi-lingual).
DONE: Basic settings/options management (ini file)
DONE: Import of DOC and DOCx, Text files, PDFs, HTML files, and Images
DONE: Storage of documents INSIDE an encrypted database (security).
DONE: Full indexation of DOC, DOCx, Text, HTML and PDF (when text content is available) content with search
DONE by FrédB: Soundex implementation (FH and US) allowing the indexation of documents in phonetic per language
DONE by FrédB: Automatic determination of a text language via a statistical analysis of n-grams
DONE: fuzzy search on top of exact search, using French OR English phonetic systems like SOUNDEX.
DONE: in house edition of DOC and DOCx files using txTextControl
DONE: Documents management (New/Edit/Delete/Export)
DONE: Users management with roles and login
DONE: Keywords management (in addition to full text indexation or INSTEAD of it, for extremely sensitive documents where we do not want the content to be indexed) allowing the concept of folders or projects
DONE: in house locking system, allowing to protect data on unattended computers while reusing the APPLICATION login/rights to unlock. Automatic and manual lock.
DONE: replication of documents (dissemination of data AND AUTOMATIC BACKUP) via WXReplication
DONE: Complex UI management (non modal) with windows' bar, touchscreen ready
TEST: Management of DOC and DOCx files using gembox document: changing the type of documents (doc/docx/pdf/txt/jpg...)
TEST: Scan of document (TWAIN)
TODO: Documents access restrictions based on Users and roles (Read/Edit/Save as New/ Save as Revision/Delete...)
TODO: OCR of documents
TODO: Import of images from disk/camera
TODO: Import of XLS and XLSx files and full text indexation
TODO: Android/Mobile document viewer
TODO: Android "scanner" and import into the main DB
TODO: Document version management
TODO: VAULT management (by double encryption of document, with per document password)
TODO: Optimization of full text indexation by background processing
TODO: Web based document viewer
TODO: Web based document uploader
And many more (perhaps :-) )
Project Guidelines (...read more...)
(2015-05-05) Replication offers a lot of advantages, but it also means that some things need to be considered... differently. Users' management and its corollary data security are clearly in that situation.
Everything below implies that:
ALL data files are encrypted and password protected. If that is not the case, talking about security is a pure joke.
For the same reason, any password information should never be stored as is, but instead a HASH of that password should be stored and used.
Direct access to the DB should be forbidden in all cases and this means:
No ODBC or other direct access to the DB for 3rd party programs (that's what webservices are for)
No direct access to the DB over the web even for YOUR programs (that's what webservices are for)
No access to the data in the hyperfile control center for anybody else BUT YOU... Anything else should be done via utilities provided in YOUR application. If the customer wants to see raw data for some obscure reason, then a window with an empty table, a query and a buildbrowsingtable are all that you need, and you CAN manage security here.
No export allowed through the AAF or printing utilities, but only through utilities coded by YOU, with a security system in place to authorize and LOG these export, and possibly report them to a security officer for control.
If you do not agree with and do not apply these principles, then do not bother reading the remaining of this article, it would be utterly pointless.
First, let 's see how this would be managed in the classical case of a centralized DB, accessed in client server.
In this case, when installing the server ad the DB, the files are created by an admin (or by the first access).
The user file is initially empty, and either a first user with a specific name/password is created automatically in that case (weak solution, as it means that this user MAY remain the same on all DBs of that program, which is NOT optimum), or the admin (anybody, really, at this point) can create a user and give it admin rights.
From now on, all other users are created by this admin (or several admins).
Security relies on the fact that nobody can delete the user file and therefore recreate the original situation where an admin user can be created. As the DB is protected on a server, this kind of operation (and the copy of the whole DB) should be impossible.
Of course, we have all heard of cases where hackers can access the server and then... But that is another story as we are trying here to devise a good INTERNAL security for your application.
Now, for the replication case, we clearly can not rely on the same fact... DBs are local to each user, and ANYBODY can delete the user files. We are basically ALWAYS in the case where access to the DB has been hacked... So, how do secure our data?
If your application creates automatically the user file if it's not here (hcreationifnotfound, option set in the project, etc), then somebody can delete it, run the program to recreate it empty, and at that point, either you are recreating the standard admin user (admin/admin, anybody), or the user can create a new record with admin rights. In both case, that's a very weak security schema.
So let's try harder: we do NOT create automatically admin/admin and we can create a new user record ONLY if ALL files are empty. That's better, but a hacker could delete all files to be in that situation, create a new admin user, then copy back all files BUT the new user file, and voila... full access to everything... So that's still a no go...
Now that we have seen all the things that cannot work, here is MY solution... Feel free to poke holes in it, as I prefer to rework the security BEFORE my application has been hacked :-). The funny thing about it is that even though it has been devised for the replication case, it is a way to improve data security in ALL cases... But as with all real security solution, it is a comprehensive effort that will work only if ALL checkboxes are checked. So, what do we need to do?
It is possible to create a first User record with admin rights (and with a manually entered password) ONLY when ALL files are empty.
Just before creating that record, the Database must be uniquely identified by a randomly assigned GUID. This information must be stored in ANOTHER FILE of the DB (in our case, it is stored in the WXReplication file)
Each record in each file must contain the DB GUID information, including the user record.
When login in, the USER DB GUID and the WXReplication DB GUID are compared. Login is possible only if they are the same.
When replicating data, records coming from other DBs (ie with a different DB GUID) must have their DB GUID set to the local DB GUID
ALL READING CODE (query -AND- hreadseek and other) MUST INCLUDE A CHECK OF THE LOCAL DB GUID. This check is done by comparing the current user DB GUID with the target data DB GUID.
If we do ALL that, let's see how we fare compared to the previous hacking attempts:
If somebody delete the user file, it is recreated empty and nobody can log in, as user creation is NOT possible (other files are NOT empty)
If somebody deletes ALL the files, they are all recreated empty, and we are in the initial install situation. Therefore, creating a user is possible. However, It will be created with a NEW and DIFFERENT DB GUID (assigned randomly)
When copying back all other files, either the hackers also copies the old WXReplication file, and the two GUIDs do not match (login impossible) or he/she copies all the files BUT the user and WXReplication files. Login IS possible, but any further data access is impossible, as once again DB GUIDs are different.
Even an internal job where an employee copies HIS valid user file to another computer to gain access to somebody else data will not work, as once again GUIDS are different.
If we accept a slightly slower login, we can even compare the user DB GUID with the DB GUID of one record of EACH FILE, and prevent login if there is any discrepancy.
And of course, applying that technique to ANY type of DB (Client Server, local, replicated or not), means that messing around with the files will not give any advantage to the hackers...
Brute force attacks and social hacking, however, are always a threat. At least, WE, as developers, do not let the keys to the kingdom on the door.
Users' management and data security (...read more...)
2015-10-13 - This is the full list of known bugs as I manages it day by day.
If you want to send me information about a problem, please send me:
- the source version (currently 0.9). If the bug is happening in a source that YOU have modified, then please test first in my original source and send me information ONLY if the bug is also happening here.
- a step by step reproduction protocol
- the FULL error text
- if the bug is relative to a specific data situation, the corresponding daataset (either DB or document or both)
Remember that if I cannot easily reproduce the bug, I cannot work on it and find it. So help me help you.
--- ALL BUGS BELOW ARE IN VERSION 0.9 ---
#1.[Corrected in next version]. In help editor, bug on up and down button. xSortOder is not properly set in some cases
#2.[Open]. From time to time, there is a hAlias error. linked to a HF context problem
#3.[Open]. When working on htm files, the quick text preview contains the file path instead of text content
Bugs (...read more...)
(2015-10-30) This is where our dreams go... To live, if everything goes according to plan.
I'm going to list here all the future development for the project, in 5 large groups: (1) Working on it... (2) Soon... (3) Later... (4) Maybe one day (but don't hold your breath... (5) Requested, but I don't see it...
Send me your requests, and I will list them here, in the group that "I" feel is appropriate (and I am allowed to change my mind too, yes I am!)
Of course, sponsors have the possibility to financially help the realization of ONE specific point, and contributors are welcome to work on anything, as long as it is coordinated with the general work and doesn't introduce incompatibilities.
(last number used=39)
Working on it...
#1[DONE-Next Version]. Exceptions management for TxTextControl: In order to intercept errors from the ActiveX, calls must be encapsulated in When Exception statements.
#2[DONE-Next Version]. Remove all FAA/Preview export options: In order to be secure, WXEDM needs to handle any export via an authorizes facility, ie not in ay way automatic/open
#3. Modify/Check unicode keys for IOS: This will have to be done both in WXEDM and WXReplication in order to support IOS in the future. There are some limitation to unicode keys in IOS and some specific settings must be used.
#4. Optimization of Search:
- Try tweaking the query
- Remove quick-preview/text memo from main file
#6. Indexation of documents in background: This will allow the user to get control of the program immediately while indexation will be done in the background by a separate facility, probably by a new replication engine.
#10. Code: Check availability of base 64 encoding across platform and complete with wl code if necessary.
#20. Scanner: Integrate test code
#8. Installer: Set up the installer to also manage the installation of gembox and txTextControl.... Create an easy to use executable. This could be done by using The same technique that the announced new v21 feature: Create a .wdl with the extra feature and call it at the end of the main installer. This will allow minimal changes when switching to v21.
#9. Init wizard: Add management of replication choices: NO (mono user), Create new group, Join a group. Group key could be a GUID displayed in base 64 (8 chars). Group key would be completed by password.
#11. CODE-WXEDM-WXREPLICATION: This is something that has to be done at the WXREPLICATION level, so it would be done in these classes and get re-incorporated into the WXReplication project. Fill the GUID field of each record with base 64 content. This could allow to:
- reduce the size of the GUID field from 32 to 8
- at the same time, pass all GUID fields in ANSI (do NOT need unicode, as characters inside are known HEXA or base 64). This would further reduce the size of each record by (64-8)=56 bytes!!!
This means that we need to check the whole replication chain in all environment to verify that ANSI values are processed correctly too. This would also solve one of the problem found for IOS.
#12. Settings: Add an option allowing to Hide languages: everything is in ONE language and in english.This will allow US users to just use it as is.
#13. Explore: See usability of text extraction using java or Command line LibreOffice (Thanks John!)
#14. Login: Add extra secure mode by clicking in moving squares (graphical) instead of entering password (key logger protection)
#15. Import: Check with the calculated CRC if another identical file is already imported. That would be an option to Check for duplicates, as users MAY want to accept duplicates in the system (generic form in different projects, by example)
#16. Check for duplicates: New general function to check entire DB for duplicates: index on CRC, if CRC equal, check byte/byte
#17. Import: Allow "replacing" (same directory, same file... ???Same CRC???)
#18. Export: Allow/propose the original file/directory
#19. Password: Improve MakeHash function to use a record related extra seed per user (calculated through everything) ( Thanks Eric!)
#21. Analysis: Add basic files for Project, company, contact, email address, phone.... Links between them. Basic forms and lists. To link a document to ANYTHING, use the main record GUID as a keyword.To search anything linked to a main record, search for the corresponding GUID/exact. Modify import accordingly.
#26. File formats: Add support of .odt, .xls, .rtf and miscellaneous files (like images, but not visible in the application)
#27. General cleaning: Correction of variables and procedures names, verification of comments, etc.
#28. In house edit: Current version still has a temporary file outside of the DB when editing a document. To improve security, modify the current code to have all files only in memory and in the DB.
#29. LibreOffice support: Add support of LibreOffice as a way of editing documents OUTSIDE of the application. This would be an option opposed to txTextControl.
#32. Extra permissions: Add extra permissions to restrict document access. By example: can saveAs, but cannot Save. Can edit only document owned (created), etc.
#5. Optimization of import: It is probably possible to gain quite a lot in optimizing the code. However, import is not done very often, so priority is low
#7. Help file management changes: (MAYBE - MORE thinking needed) Help file should not contain a local DB GUID field. This would allow to exchange help files between sites easily. However, if exchanging help files, what would happen with replication of said file and user generated content. Some thinking to do here.
#22. Help: Allow to link any document(s) of the DB to any topic/chapter/page.
#23. Help: Modify help to support RTF (images inside) (option, on the web and mobile, only text is displayed)
#24. Sandbox: Add Sandbox implementation for basic files AND documents.
#25. Customization: In all basic files (project, company, etc), add a text memo field for ini like extension and add management of customization to each file/form/.list
#30. MS Office support: Add support of MS office as a way of editing documents OUTSIDE of the application. This would be an option opposed to txTextControl.
#31. OCR: Add support of an OCR solution to the scanner and import features.
#33. Revisions: Add support of revisions/versions of the same document.
#35. Camera: Add import from camera/phone (from DCIM directory or other directories, with automatic recognition of the phone camera if possible
#36. Android document viewer and scanner: Would use the replication system to get a list of authorized document and either the documents themselves or an image of them when needed or in advance. Use the phone camera to scan document and add comment and send them via replication. Create text documents/notes linked to projects, companies, etc
#37. IOS document viewer and scanner: Would use the replication system to get a list of authorized document and either the documents themselves or an image of them when needed or in advance. Use the phone camera to scan document and add comment and send them via replication. Create text documents/notes linked to projects, companies, etc
#38. WEB document viewer and uploader: Would use the replication system to get a list of authorized document and either the documents themselves or an image of them when needed or in advance. Allow to upload document and add comment and send them via replication. Create text documents/notes linked to projects, companies, etc
#39. Vault: The files are already encrypted in the database. the VAULT function would allow to do a SECONDARY encryption of the file and its content BEFORE being stored in the DB. Encryption key would be specific to EACH file and not stored anywhere (lost key=lost document). Content of the document put in vault would NOT be indexed for retrieval, so only MANUAL KEYWORDS/folder would allow access to document.
Maybe one day (but don't hold your breath)...
#34. Scenarios: Add support of scenarios to manage the production of documents. By example: when a secretary (role) create form X, this form enters the system and a supervisor (role) is alerted that he needs to verify and approve the document. If document is approved, it goes to head of service for signature, if not if goes back to secretary for corrections. Only when signed would the document be printable/sendable. This system would allow to define these scenario, choose the template to use, the naming convention, the links to main files, the information automatically "fused" into the template, the permissions of each user or type of user, the different steps needed, the conditions to go to the next steps, the organigram of steps possible/required, and the options at each step.
Requested, but frankly I don't see it...
In the future (...read more...)
WinDev, WebDev and WinDev Mobile consulting
Create an account!
RC4 for WX
BEFORE YOU DOWNLOAD
All the things you need to know
About the site
DPS par DataTale
Enter your user credentials below
I forgot my password
Create an account, it's free!
Message in the cell