| Version 15 (modified by os, 14 years ago) (diff) |
|---|
OpenSubtitles v2 draft specification
Introduction
Programming in python use PEP-8 code practices, another good basics
Subtitles section
We are trying to avoid duplicate subtitles as much as possible, so in ideal world, there should be only one subtitle for many releases. We try to approach this by using SubLib. So, in system there will be saved only 1 subtitle for each version of movie: [Matrix], [Matrix - Extended Cut], [Matrix - Directors Cut] and so on. This is ideal situation and current world is not ideal.
We know, there are many versions of rip, so in ideal world, there should be just some rules, how to change original subtitles to fit them to the movie version. For example:
- Take [Matrix] subtitles id 123456
- Change frame-rate from 25 FPS to 23.978 FPS
- Add 3 seconds to the beginning
- Cut subtitles at 1 hour 25 minutes 45 seconds 78 milliseconds and make 2 files
With these rules we can represent any version, and hopefully any needs for movie. One big advantage of this system is wiki-style of subtitle editing, so, you look movie, you will find some bad translation or typo, you login to the site and edit these subtitles online (or using some program implementing our API). All these changes will be present for the all versions.
So in the system will be one master subtitle for each version of movie (original version, directors cut). This will be beginning and in database we will got rules, how to "change" it for different movie rips. So, master subtitle is present in database and by analyzing timestamps of other uploaded subtitles we got rules for re-timing it to the another movie rips, that's theory.
Wiki editing - users should be able to edit and translate subtitles online, with versioning system. All changes will be tracked, so in the final we will have in system how many changes was done by each user.
Subtitles export - subtitles in the system are saved as metadata, so user can choose any subtitle format as he want.
Realtime Re-timing, cutting - SubLib supports re-timing, cutting, "moving" subtitles, so this should be done also online and via API.
With subtitles store its encoding. Use charset detection For language detection use TextCat, Python implementation, langdet, google translate python lib
Movie section
Implement more than one website for movies, now is implemented only imdb.com python wrapper, which is not bad, but they don't provide any official API access to their database. That's why there is need to implement sites like TheMovieDB.org python wrapper here and TheTVDB.com python wrapper. So support 3 sites, imdb.com as last, when 2 other fails (?). Mark if movie is TV Series, add Season and Episode
Movie hashing - there is little need for stronger hash, which need some research, how to done it properly, because current implementation (CRC64) is weak and can lead to collisions in future (so far there was no collisions). Ideally, system should be coded for more kind of hashes. I think wrong idea is put into hash itself information such is movielength, fps, dimensions of movie and such. Hash should be only file dependent, for example first and last 128kb (sha1), and filesize together hashed (sha1).
Media information - in database there is need to save such information, but problem is implementation can be different in programs. The most important is FPS.
User Section
Registration - simple as possible - username, email, password, possible login using social sites, openid and so on (rpxnow.com).
Groups and permissions - similar like in current version of opensubtitles. There are permissions, groups got 1 or many permissions, and user can belong to 1 or more groups. Django got good resources of libs for this.
Translator Groups
There should be some good support for subtitle translators and their groups. This need more research how to done it properly.
- rating system
- groups rating
- stats
Upload section
Website upload: user should upload files first and then fill information needed. Why? Because we can help him to fill these informations - we can analyze subtitles for language, so analyzed subtitle language will be selected. Duplicates should be avoided, or very similar subtitles, so thats why it is wise to let user just select subtitles and upload and then fill needed information.
Request section
Subtitles should be requested as now. There should be progress in work, if subtitles for movie exists or not, and bidding (donating) for translation, so translators could choose their subtitles according money offer also.
Website Translation
Current system of website translation is OK. Need more comments from danger how it will be done. Just notes:
- only some users will have access to translate webpage
- there should be mails sent each friday morning, if there are some untranslated items, or items needs to be translated
- mark in web translation module if some item needs to be retranslated (original text changes)
API Access
Only registered useragents will have API access by using their key. API should be provided by different standards such as XML-RPC, REST, JSON...good example of API is on TheMovieDB. API versioning is a must.
Caching
Software specification
Lighttpd as http server, running FastCGI Postgre SQL as database server, Python as programming language, Django as framework
- Django-Sphinx
- Web Services
MongoDB for caching of subtitles
Memcache for memory caching Sphinx-search for fulltext search
Other notes
- Ratings should just be upvote and downvote.
- Multiple-Subtitle TryUpload support, with response for all requests sent.
- Multiple Movie subtitle upload support (just allowing upload of more than one CD array)
- Official support for subtitle upload without matched movie
- Simple multiple deletion of subtitles
- Community picked "best-version" of subtitle
- maximum two subtitles per language per release
- preview of subtitles will be AJAX style, fetched from cached subtitles
Database Design
Tables name and columns name are not final and will be changed.
Version 1
Not added tables for:
- website translation
- permission/groups users
Notes:
- subs_movie_data -> store raw python-serialized info from other website api response
Discussion
where to store subtitle files ?
It seems the best thing is to store METADATA of subtitle files to Database, because how users can translate subtitles online, when it is not in database? For Caching of subtitles (so subtitles doesn't have to be each time REGENERATED) we can use MongoDB
using of API key for useragent is good idea ?
if somebody wants hack it, he will just look for API key in other app and thats it. So what is difference to have enabled useragents and API keys?
