mirror of https://github.com/tgragnato/magnetico
Update README.md
This commit is contained in:
Tommaso Gragnato
parent
0b73026cd5
commit
ca10d35656
90
README.md
90
README.md
|
|
@ -1,32 +1,21 @@
|
|||
> # Archival Notice
|
||||
>
|
||||
> Hello! 👋 I've been working on **magnetico** since 2017 (less so in the recent years) and seeing so many people interested in it and using it has been a great source of joy and pride for me. However, I have decided that it is for the best to acknowledge and admit openly that I no longer have as much time as I did in high school, nor any willingness to spend the little time I now have by working on **magnetico** or any other [free software](https://en.wikipedia.org/wiki/Free_software) in general, except what I think is [the most important problem in the world](http://www.aaronsw.com/weblog/productivity#:~:text=not%20working%20on-,the%20most%20important%20problem%20in%20the%20world,-\)%20but%20each%20little) that I can work on at that moment.
|
||||
>
|
||||
> Fork it, improve it, ship it; keep up the good fight against scarcity. ☀️
|
||||
>
|
||||
> Bora <bora at [boramalper.org](https://boramalper.org/)>
|
||||
|
||||
# magnetico
|
||||
*Autonomous (self-hosted) BitTorrent DHT search engine suite.*
|
||||
|
||||
[](https://gitter.im/magnetico-dev/magnetico-dev) [](https://github.com/boramalper/magnetico/actions) [](https://bestpractices.coreinfrastructure.org/projects/1029)
|
||||
|
||||
|
||||
magnetico is the first autonomous (self-hosted) BitTorrent DHT search engine suite that is *designed
|
||||
for end-users*. The suite consists of two packages:
|
||||
magnetico is the first autonomous (self-hosted) BitTorrent DHT search engine suite that is *designed for end-users*. The suite consists of two packages:
|
||||
|
||||
- **magneticod:** Autonomous BitTorrent DHT crawler and metadata fetcher.
|
||||
- **magneticow:** Lightweight web interface for magnetico.
|
||||
- **magneticod:** is the daemon that crawls the BitTorrent DHT network in the background to discover info hashes and fetches metadata from the peers.
|
||||
- **magneticow:** is a lightweight web interface to search and to browse the torrents that its counterpart discovered.
|
||||
|
||||
Both programs, combined together, allows anyone with a decent Internet connection to access the vast
|
||||
amount of torrents waiting to be discovered within the BitTorrent DHT space, *without relying on any
|
||||
central entity*.
|
||||
Both programs, combined together, allows anyone with a decent Internet connection to access the vast amount of torrents waiting to be discovered within the BitTorrent DHT space, *without relying on any central entity*.
|
||||
|
||||
**magnetico** liberates BitTorrent from the yoke of centralised trackers & web-sites and makes it
|
||||
*truly decentralised*. Finally!
|
||||
|
||||
## Features
|
||||
- Easy installation & minimal requirements:
|
||||
- [Pre-compiled static binaries](https://github.com/boramalper/magnetico/releases) and [Docker images](https://hub.docker.com/u/boramalper) are provided.
|
||||
- Easy to build golang static binaries.
|
||||
- Root access is *not* required to install or to use.
|
||||
- Near-zero configuration:
|
||||
- Both programs work out of the box, and **magneticow** can be used without a web-server too.
|
||||
|
|
@ -46,10 +35,8 @@ central entity*.
|
|||
getting on your way.
|
||||
|
||||
### Screenshots
|
||||
*Click on the images to view full-screen.*
|
||||
|
||||
<!-- Use https://www.tablesgenerator.com/markdown_tables -->
|
||||
|  |  |  |
|
||||
|  |  |  |
|
||||
|:-------------------------------------------------------------------------------------------------------------------------------------------------------:|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------:|:---------------------------------------------------------------------------------------------------------------------------------------------:|
|
||||
| __The Homepage__ | __Searching for torrents__ | __Viewing the metadata of a torrent__ |
|
||||
|
||||
|
|
@ -60,66 +47,3 @@ other peers (trackers). Introduction of DHT (distributed hash table) eliminated
|
|||
trackers, allowing peers to discover each other through other peers and to fetch metadata from the
|
||||
leechers & seeders in the network. **magnetico** is the finishing move that allows users to search
|
||||
for torrents in the network, hence removing the need for centralised torrent websites.
|
||||
|
||||
## Installation Instructions
|
||||
> **WARNING:**
|
||||
>
|
||||
> **magnetico** is still under active construction, and is considered *alpha* software. Please
|
||||
> use **magnetico** suite with care and follow the installation instructions carefully to install
|
||||
> it & secure the installation. Feel perfectly free to send bug reports, suggestions, or whatever
|
||||
> comes to your mind to send to us through GitHub or personal e-mail.
|
||||
|
||||
> **WARNING:**
|
||||
>
|
||||
> Do NOT clone the [repository](https://github.com/boramalper/magnetico) to install **magnetico**,
|
||||
> as it is never meant to be stable (except
|
||||
> [releases](https://github.com/boramalper/magnetico/releases) of course).
|
||||
|
||||
1. Install **magneticod** first by following its [installation instructions](cmd/magneticod/README.md).
|
||||
2. Install **magneticow** afterwards by following its
|
||||
[installation instructions](cmd/magneticow/README.md).
|
||||
|
||||
### Docker
|
||||
|
||||
Run **magneticod** and **magneticow** with:
|
||||
|
||||
``` bash
|
||||
make docker
|
||||
```
|
||||
|
||||
It will run magnetico from already built image on [Docker Hub](https://hub.docker.com/u/boramalper)!
|
||||
|
||||
You should access magneticow at <http://localhost:8080>.
|
||||
|
||||
To build fresh images from source, first run:
|
||||
|
||||
``` bash
|
||||
make image
|
||||
```
|
||||
|
||||
Then run `make docker`. It ensures you run updated images of magnetico.
|
||||
|
||||
## License
|
||||
|
||||
All the code is licensed under AGPLv3, unless stated otherwise specifically. See `COPYING` for
|
||||
details.
|
||||
|
||||
## Donations
|
||||
### Patreon
|
||||
https://www.patreon.com/boramalper
|
||||
|
||||
### PayPal
|
||||
https://paypal.me/boramalper
|
||||
|
||||
### Cryptocurrencies (Coinbase)
|
||||
- **BTC:** `3BLWjamWug3QQzcDDGwYLwuCqJyjcfYJB8`
|
||||
- **LTC:** `MRWX5SGCF7EvN15gpzT5b3KQD3Z91gH8qi`
|
||||
- **BCH:** `qqn07a58hax9l8pckq9j8ys6dsh2cnu4rsyztw2kj9`
|
||||
- **ETH:** `0xe5A8e80bAA6129DF7eBB1B5302F9e2Ef4C6f6E62`
|
||||
- **ETC:** `0x8964EcC86eaf043Bff2CdfE875E73D8095c26a58`
|
||||
|
||||
----
|
||||
|
||||
Dedicated to Cemile Binay, in whose hands I thrived.
|
||||
|
||||
Bora M. ALPER <bora@boramalper.org>
|
||||
|
|
|
|||
|
|
@ -1,74 +0,0 @@
|
|||
# magneticod
|
||||
*Autonomous BitTorrent DHT crawler and metadata fetcher.*
|
||||
|
||||
**magneticod** is the daemon that crawls the BitTorrent DHT network in the background to discover info hashes and
|
||||
fetches metadata from the peers.
|
||||
|
||||
## Installation
|
||||
|
||||
### Requirements
|
||||
- Decent Internet access (IPv4)
|
||||
|
||||
**magneticod** uses UDP protocol to communicate with the nodes in the DHT network, and TCP to communicate with the
|
||||
peers while fetching metadata. **Please make sure you have a healthy connection;** you can confirm this by checking at
|
||||
the *connection status indicator* of your BitTorrent client: if it does not indicate any error (*e.g.* a misconfigured NAT),
|
||||
**magneticod** should just work fine.
|
||||
|
||||
### Installing the Pre-Compiled Static Binary
|
||||
You can find the latest pre-compiled static binaries on [GitHub](https://github.com/boramalper/magnetico/releases)
|
||||
for versions from v0.7.0 onwards.
|
||||
|
||||
### Installing the Docker Image
|
||||
Docker images are provided on [Docker Hub](https://hub.docker.com/r/boramalper/magnetico/tags/) at
|
||||
the repository `boramalper/magnetico`. Images are tagged as `d-vMAJOR.MINOR.PATCH`.
|
||||
|
||||
## Setup
|
||||
1. (Optional, **requires root**) Disable iptables for a specified port:
|
||||
|
||||
```bash
|
||||
iptables -I OUTPUT -t raw -p udp --sport PORT_NUMBER -j NOTRACK
|
||||
iptables -I PREROUTING -t raw -p udp --dport PORT_NUMBER -j NOTRACK
|
||||
```
|
||||
|
||||
This is to prevent excessive number of ``EPERM`` "Operation not permitted" errors, which also has a negative impact
|
||||
on the performance.
|
||||
|
||||
## Usage
|
||||
### Database
|
||||
**magneticod** is designed to be able to use different database engines to store its data, but
|
||||
currently only SQLite 3 and PostgreSQL 9+ are supported.
|
||||
|
||||
#### SQLite
|
||||
|
||||
The database file can be found in:
|
||||
|
||||
- **On Linux**
|
||||
|
||||
~/.local/share/magneticod/
|
||||
|
||||
**magneticod** uses write-ahead logging (WAL) for its database, so there might be multiple
|
||||
files while it is operating, but ``database.sqlite3`` is *the database*.
|
||||
|
||||
#### More engines (PostgreSQL and others)
|
||||
|
||||
You can read about other supported persistence engines [here](pkg/README.md).
|
||||
|
||||
### Using the Docker Image
|
||||
You need to mount
|
||||
|
||||
- the data directory (`~/.local/share/magneticod/` on Linux, see the previous sections)
|
||||
- the configuration file at (`~/.config/magneticod/configuration.toml` on Linux, see the previous sections)
|
||||
|
||||
hence run:
|
||||
|
||||
```bash
|
||||
docker run -it --rm \
|
||||
-v ~/.local/share/magneticod:/root/.local/share/magneticod/ \
|
||||
-v ~/.config/magneticod/configuration.toml:/root/.config/magneticod/configuration.toml \
|
||||
boramalper/magneticod
|
||||
```
|
||||
|
||||
### Remark About the Network Usage
|
||||
**magneticod** does *not* have any built-in rate limiter *yet*, and it will literally suck the hell out of your
|
||||
bandwidth. Unless you are running **magneticod** on a separate machine dedicated for it, you might want to consider
|
||||
starting it manually only when network load is low (e.g. when you are at work or sleeping at night).
|
||||
|
|
@ -4,17 +4,17 @@
|
|||
**magneticow** is a lightweight web interface to search and to browse the torrents that its counterpart (**magneticod**)
|
||||
discovered.
|
||||
|
||||
See the list of [alternative front-ends](https://github.com/boramalper/magnetico/wiki/Related-Projects#alternative-front-ends)
|
||||
See the list of [alternative front-ends](https://github.com/tgragnato/magnetico/wiki/Related-Projects#alternative-front-ends)
|
||||
developed by the community if you need something more advanced or different.
|
||||
|
||||
## Installation
|
||||
### Installing the Pre-Compiled Static Binary
|
||||
You can find the latest pre-compiled static binaries on [GitHub](https://github.com/boramalper/magnetico/releases)
|
||||
You can find the latest pre-compiled static binaries on [GitHub](https://github.com/tgragnato/magnetico/releases)
|
||||
for versions from v0.7.0 onwards.
|
||||
|
||||
### Installing the Docker Image
|
||||
Docker images are provided on [Docker Hub](https://hub.docker.com/u/boramalper) at
|
||||
the repositories `boramalper/magneticod` and `boramalper/magneticow`.
|
||||
the repositories `tgragnato/magneticod` and `tgragnato/magneticow`.
|
||||
|
||||
## Setup
|
||||
### Configuration
|
||||
|
|
@ -83,7 +83,7 @@ hence run:
|
|||
docker run -it --rm \
|
||||
-v ~/.local/share/magneticod:/root/.local/share/magneticod/ \
|
||||
-v ~/.config/magneticow/configuration.toml:/root/.config/magneticow/configuration.toml \
|
||||
boramalper/magneticow
|
||||
tgragnato/magneticow
|
||||
```
|
||||
|
||||
Using Docker, the default username & password is `magnetico` and `magnetico`.
|
||||
|
|
@ -128,4 +128,4 @@ Using Docker, the default username & password is `magnetico` and `magnetico`.
|
|||
|
||||
**magneticow** offers a REST-ful HTTP API that is capable of everything the web interface can do.
|
||||
|
||||
See the [API documentation on Swaggerhub](https://app.swaggerhub.com/apis/boramalper/magneticow-api/v0.1).
|
||||
See the [API documentation on Swaggerhub](https://app.swaggerhub.com/apis/tgragnato/magneticow-api/v0.1).
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 14 KiB |
|
Before Width: | Height: | Size: 89 KiB After Width: | Height: | Size: 89 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 133 KiB |
|
|
@ -1,68 +0,0 @@
|
|||
# magnetico/pkg
|
||||
[](https://godoc.org/github.com/boramalper/magnetico)
|
||||
|
||||
- The most significant package is `persistence`, that abstracts access to the
|
||||
magnetico databases with different engines (currently, SQLite, stdout and partly PostgreSQL).
|
||||
|
||||
**For REST-ful magneticow API, see [https://app.swaggerhub.com/apis/boramalper/magneticow-api/](https://app.swaggerhub.com/apis/boramalper/magneticow-api/).**
|
||||
|
||||
## PostgreSQL database engine (only `magneticod` part implemented)
|
||||
|
||||
PostgreSQL database engine uses [PostgreSQL](https://www.postgresql.org/) to store indexed
|
||||
torrents. It's more performant and flexible than SQLite but requires additional software configuration.
|
||||
|
||||
**WARNING:** `pg_trgm` extension required. You need to enable it for your database before starting `magneticod`:
|
||||
|
||||
```postgresql
|
||||
CREATE EXTENSION pg_trgm;
|
||||
```
|
||||
|
||||
Engine usage example:
|
||||
|
||||
```shell
|
||||
magneticod --database=postgres://username:password@127.0.0.1:5432/database?schema=custom_schema_name&sslmode=disable
|
||||
```
|
||||
|
||||
See [pgx repository](https://github.com/jackc/pgx/blob/master/stdlib/sql.go)
|
||||
for more examples.
|
||||
|
||||
Optional parameter `schema` was added to choose which schema will be used to store magnetico tables,
|
||||
sequences and indexes.
|
||||
|
||||
## Beanstalk MQ engine for magneticod
|
||||
|
||||
[Beanstalkd](https://beanstalkd.github.io/) is very lightweight and simple MQ server implementation.
|
||||
You can use it to organize delivery of the indexed data to your application.
|
||||
|
||||
Use `beanstalk` URL schema to connect to beanstalkd server. For example:
|
||||
|
||||
```shell
|
||||
magneticod --database=beanstalkd://127.0.0.1:11300/magneticod_tube
|
||||
```
|
||||
|
||||
Don't forget to [set](https://linux.die.net/man/1/beanstalkd) binlog persistence, change maximum job size
|
||||
and `fsync()` period to be able to reliably save torrents with a large number of files:
|
||||
|
||||
```shell
|
||||
# Example settings (may not work for you)
|
||||
beanstalkd -z 1048560 -b /var/lib/beanstalkd -f 2400000
|
||||
```
|
||||
|
||||
For job data example see `stdout` engine documentation below as `beanstalk` engine uses the same format.
|
||||
|
||||
## Stdout Dummy Database Engine for magneticod
|
||||
|
||||
Stdout dummy database engine for **magneticod** prints a new [JSON Line](http://jsonlines.org/)
|
||||
for each discovered torrent so that you can pipe the stdout of **magneticod** into some other
|
||||
program to build your own pipelines on the fly!
|
||||
|
||||
**Example Output**
|
||||
|
||||
```json
|
||||
{"infoHash":"f84b51f0d2c3455ab5dabb6643b4340234cd036e","name":"Big_Buck_Bunny_1080p_surround_frostclick.com_frostwire.com","files":[{"size":928670754,"path":"Big_Buck_Bunny_1080p_surround_FrostWire.com.avi"},{"size":5008,"path":"PROMOTE_YOUR_CONTENT_ON_FROSTWIRE_01_06_09.txt"},{"size":3456234,"path":"Pressrelease_BickBuckBunny_premiere.pdf"},{"size":180,"path":"license.txt"}]}
|
||||
```
|
||||
|
||||
> **WARNING:**
|
||||
>
|
||||
> Please beware that the schema of the object (dictionary) might change in backwards-incompatible ways
|
||||
> in the future; although I'll do my best to ensure it won't happen.
|
||||
Loading…
Reference in New Issue