Time flies faster when you are having fun! I didn't believe it back then. But now I do after experiencing it. It couldn't have been more accurate that here I am writing my concluding blog of the GSoC 2020: The Linked Commons series when I just started enjoying things.
In this post, I will give a brief overview of the linked commons and my GSoC contributions. It was an exciting journey, and I loved working on this project.
Before I begin any further just for ritual, let me share a one liner on what The Linked Commons is, although I highly recommend reading the other posts in this series, who knows you might join our team.😉
The CC catalog data visualization or linked commons is a web application which finds and explores relationships between Creative Commons licensed content on the web.
My primary contributions to the project during the GSoC timeline were threefold.
Firstly, revamp the design and migrate the project to react.js for a fast and scalable rendering performance.
Secondly, add graph filtering methods and scale the data to enable users to visualize massive data more efficiently.
At last, make the developer onboarding easy by dockerizing the project and bring more application portability.
GSoC Work Product
The live version of the linked commons can be found here. You can interact with it and "explore the creative commons in graphs".
If you wish to access the raw or filtered data, then here is a brief documentation of our new API.
URL: /api/graph-data Method: GET Description: Returns a randomized graph having around 500 nodes and links.
URL: /api/graph-data/?name={node_name} Method: GET Description: Returns the filtered graph with a set of nodes which are either immediate neighbours to {node_name} in the original graph or the transpose graph.
URL: /api/suggestions/?q={query} Method: GET Description: Returns a set of nodes which contains the {query} pattern in their nodeid
Demo
My Code Contributions
Repository: https://github.com/cc-archive/cccatalog-dataviz/
Commits: https://github.com/cc-archive/cccatalog-dataviz/commits/master
Contributors: https://github.com/cc-archive/cccatalog-dataviz/graphs/contributors
- Migrated the frontend to a web application using React.js for smooth rendering performance.
- Add client-side graph filtering method to enable users to interact with the loaded graph.
- We realized that the client-side graph filtering method is not very scalable. This PR adds the basic structure for the backend server and adds server-side graph filtering logic.
- Added a parser to convert the input JSON file from
{nodes:[], links:[]}schema to the distance list format.
- It revamped the design of the frontend.
- Added both primary light theme and secondary dark theme
Add node suggestions feature #35
- Added query autocomplete feature, to enable users to explore all the nodes in the database.
- This functionality aims to minimize the number of misspelt filtering tries from the client.
- Refer to this blog for the motivation and detailed report on why we added autocomplete aka node suggestions feature.
- Optimizes the build-dB-script to run efficiently on the larger and newer dataset of the cc-catalog.
- Added the basic form of the randomized graph filtering method.
- Refer to this blog for a piece of detailed information on data update.
Database upgrade and core enhancements #39
- It upgrades the primary database from shelve to MongoDB for higher performance.
- Dockerizes the frontend and backend for both dev and prod environments for higher application portability.
- Fixes common UI bugs, updates the frontend design and enhances the mobile and smaller devices experience with the linked commons
- Modularizes and updates the code documentation
What’s Next?
Throughout this internship period, we, the Linked Commons team, aimed to make this version the best among all. But there is still scope for improvement.
Just to give you some insights; currently, the complete graph contains 235k nodes and 4.14million links. During the preprocessing, we dropped a lot of the nodes. Additionally, we removed more than 3 million nodes which didn't have cc_licenses information. So, in general, the current version shows only those nodes which are soundly linked with other domains and their licenses information is available. To give a complete picture of the massive "cc-catalog", the linked commons need to "gird up his loins".
After seeing the tremendous potential it has, I will undoubtedly continue working on it and help the linked commons in this quest. ⚡
Ending Note
In the end, I would like to thank my mentors Maria and Brent, for their unconditional guidance throughout this internship period. The insights I got from them will truly help me in the days to come.
Special thanks to Francisco, Anna and Kriti for the awesome brainstorming ideas in the UX meet which helped us build an increment superior version of the Linked Commons.
It is not the end, rather a new beginning. Cheers! 🚀🚀🚀