2023-11-27 20:04:15 +00:00
|
|
|
# WIP: descargador masivo de datos públicos
|
|
|
|
|
|
|
|
|
|
require [Node.js](https://nodejs.org) y [pnpm](https://pnpm.io/)
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
pnpm install
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## correr
|
|
|
|
|
|
|
|
|
|
```
|
2023-11-29 00:00:19 +00:00
|
|
|
# descargar portal datos.gob.ar
|
|
|
|
|
pnpm run run https://datos.gob.ar/data.json
|
|
|
|
|
# guarda en data/datos.gob.ar
|
|
|
|
|
|
|
|
|
|
# descargar todos los portales conocidos
|
|
|
|
|
pnpm run run
|
|
|
|
|
# guarda en data/*
|
2023-11-27 20:04:15 +00:00
|
|
|
```
|
2023-11-28 03:44:47 +00:00
|
|
|
|
2023-11-28 21:38:40 +00:00
|
|
|
## contenedor
|
|
|
|
|
|
|
|
|
|
```
|
2023-11-28 22:58:17 +00:00
|
|
|
docker run --rm -it -v ./data:/data gitea.nulo.in/nulo/transicion-desordenada-diablo/downloader
|
2023-11-28 21:38:40 +00:00
|
|
|
# descarga datos.gob.ar
|
|
|
|
|
```
|
|
|
|
|
|
2023-11-28 03:44:47 +00:00
|
|
|
## formato de repo guardado
|
|
|
|
|
|
|
|
|
|
- `{dominio de repo}`
|
|
|
|
|
- `data.json`
|
2023-11-28 22:58:00 +00:00
|
|
|
- `errors.jsonl`: archivo con todos los errores que se obtuvieron al intentar descargar todo.
|
2023-11-28 03:44:47 +00:00
|
|
|
- `{identifier de dataset}`
|
|
|
|
|
- `{identifier de distribution}`
|
|
|
|
|
- `{fileName (o, si no existe, identifier de distribution)}`
|
