Community contributed diagram of Ente's Replication Process
+Community contributed diagram of Ente's replication process
> [!IMPORTANT] -> As of now, Replication works only if all the 3 storage type -> needs are fulfilled (1 Hot, 1 Cold and 1 Glacier Storage). +> +> As of now, replication works only if all the 3 storage type needs are +> fulfilled (1 hot, 1 cold and 1 glacier storage). > > [Reference](https://github.com/ente-io/ente/discussions/3167#discussioncomment-10585970) -If you're wondering why there are 3 buckets on MinIO UI - that's because our -production instance uses these to perform [replication](https://ente.io/reliability/). +If you're wondering why there are 3 buckets on the MinIO UI - that's because our +production instance uses these to perform +[replication](https://ente.io/reliability/). If you're also wondering about why the bucket names are specifically what they are, it's because that is exactly what we are using on our production instance. @@ -72,10 +78,10 @@ We use `b2-eu-cen` as hot, `wasabi-eu-central-2-v3` as cold (also the secondary and `scw-eu-fr-v3` as glacier storage. As of now, all of this is hardcoded. Hence, the same hardcoded configuration is applied when you self host Ente. -In a Self hosted Ente Instance replication is turned off by default. -When replication is turned off, only the first bucket (`b2-eu-cen`) is used, -and the other two are ignored. Only the names here are specifically fixed, but -in the configuration body you can put any other keys. It does not have any relation +In a self hosted Ente instance replication is turned off by default. When +replication is turned off, only the first bucket (`b2-eu-cen`) is used, and the +other two are ignored. Only the names here are specifically fixed, but in the +configuration body you can put any other keys. It does not have any relation with `b2`, `wasabi` or even `scaleway`. Use the `s3.hot_storage.primary` option if you'd like to set one of the other @@ -85,23 +91,23 @@ predefined buckets as the primary bucket. > [!NOTE] > -> If you need to configure SSL, for example if you're running over the internet, -> you'll need to turn off `s3.are_local_buckets` (which disables SSL in the -> default starter compose template). +> If you need to configure SSL, you'll need to turn off `s3.are_local_buckets` +> (which disables SSL in the default starter compose template). > Disabling `s3.are_local_buckets` also switches to the subdomain style URLs for -the buckets. However, not all S3 providers support these, in particular, minio -does not work with these in default configuration. So in such cases you'll -also need to then enable `s3.use_path_style_urls`. +the buckets. However, not all S3 providers support these. In particular, MinIO +does not work with these in default configuration. So in such cases you'll also +need to enable `s3.use_path_style_urls`. ## Summary Set the S3 bucket `endpoint` in `credentials.yaml` to a `yourserverip:3200` or -some such IP/hostname that accessible from both where you are running the Ente -clients (e.g. the mobile app) and also from within the Docker compose cluster. +some such IP / hostname that is accessible from both where you are running the +Ente clients (e.g. the mobile app) and also from within the Docker compose +cluster. -#### Example +### Example An example `museum.yaml` when you're trying to connect to museum running on your computer from your phone on the same WiFi network: @@ -117,49 +123,44 @@ s3: bucket: b2-eu-cen ``` -## FAE (Frequently Answered Errors) +## Frequently encountered errors -Here are some Frequently Answered Errors from the Community Chat with the reasoning -for a particular error and its potential fix. +Here are some errors our community members frequently encountered with the +context and potential fixes. -In most situations, the problem turns out to be some minute mistakes or misconfigurations -on the users end and that turns out to be the bottleneck of the whole problem. -Please make sure to `reverse_proxy` Museum to a domain as well as check your S3 -Credentials and whole config for any minor mis-configurations. +In most situations, the problem is because of a minor mistakes or +misconfiguration. Please make sure to `reverse_proxy` museum to a domain and +check your S3 credentials and whole configuration file for any minor +misconfigurations. -It is also suggested that the user setups Bucket CORS on MinIO or any external -S3 Bucket they are connecting to. To setup Bucket CORS, help yourself by upload -[this](https://help.ente.io/self-hosting/guides/external-s3#_5-fix-potential-cors-issue-with-your-bucket). +It is also suggested that the user setups bucket CORS on MinIO or any external +S3 service provider they are connecting to. To setup bucket CORS, please [read +this](/self-hosting/guides/external-s3#_5-fix-potential-cors-issue-with-your-bucket). ### 403 Forbidden If museum (`2`) is able to make a network connection to your S3 bucket (`3`) but -uploads are still failing, it could be a credentials or permissions issue. A -telltale sign of this is that in the museum logs you can see `403 Forbidden` +uploads are still failing, it could be a credentials or permissions issue. + +A telltale sign of this is that in the museum logs you can see `403 Forbidden` errors about it not able to find the size of a file even though the corresponding object exists in the S3 bucket. -To fix these, you should ensure the following: +This could be because -1. The bucket CORS rules do not allow museum to access these objects. - - For uploading files from the browser, you will need to currently set - allowedOrigins to "\*", and allow the "X-Auth-Token", "X-Client-Package" - headers configuration too. - [Here is an example of a working configuration](https://github.com/ente-io/ente/discussions/1764#discussioncomment-9478204). +1. The bucket CORS rules do not allow museum to access these objects. For + uploading files from the browser, you will need to set `allowedOrigins` to + `*`, and allow the `X-Auth-Token`, `X-Client-Package` headers configuration + too. [Here is an example of a working + configuration](https://github.com/ente-io/ente/discussions/1764#discussioncomment-9478204). 2. The credentials are not being picked up (you might be setting the correct - creds, but not in the place where museum picks them from). + credentials, but not in the place where museum reads them from). -### Mismatch in File Size +### Mismatch in file size -The "Mismatch in File Size" error mostly occurs in a situation where the client (`1`) -is re-uploading a file which is already in the bucket with a different File Size. The -reason for re-upload could be anything including Network issue, sudden killing of app -before the upload is complete and etc. +The "Mismatch in file size" error mostly occurs in a situation where the client +(`1`) is re-uploading a file which is already in the bucket with a different +file size. The reason for re-upload could be anything including network issue, +sudden killing of app before the upload is complete and etc. -This is also one of Museums (`2`) Validation Checks for the size of file being -re-uploaded from the client to the size of the file which is already -uploaded to the S3 Bucket. - -In most case, it is very unlikely that this error could be a cause of some mistake in -the configuration or Browser/Bucket CORS. diff --git a/docs/docs/self-hosting/index.md b/docs/docs/self-hosting/index.md index 016504cc20..c0eef22f2c 100644 --- a/docs/docs/self-hosting/index.md +++ b/docs/docs/self-hosting/index.md @@ -10,34 +10,32 @@ the same code we use for our own cloud service. > [!TIP] > -> To get some context, you might find our -> [blog post](https://ente.io/blog/open-sourcing-our-server/) announcing the -> open sourcing of our server useful. +> You might find our [blog post](https://ente.io/blog/open-sourcing-our-server/) +> announcing the open sourcing of our server useful. +## System requirements -## System Requirements - -The server has minimal resource requirements, running as a lightweight Go binary -with no server-side ML. It performs well on small cloud instances, old laptops, -and even [low-end embedded devices](https://github.com/ente-io/ente/discussions/594) -reported by community members. Virtually any reasonable hardware should be sufficient. +The server has minimal resource requirements, running as a lightweight Go +binary. It performs well on small cloud instances, old laptops, and even +[low-end embedded devices](https://github.com/ente-io/ente/discussions/594). ## Getting started -Execute the below one-liner command in your terminal to setup Ente on your system. +Run this command on your terminal to setup Ente. ```sh sh -c "$(curl -fsSL https://raw.githubusercontent.com/ente-io/ente/main/server/quickstart.sh)" ``` -The above `curl` command is a simple shell-script, which pulls the docker images, -creates a directory `my-ente` in the current working directory and starts all the -containers required to run Ente on your system. +The above `curl` command pulls the Docker image, creates a directory `my-ente` +in the current working directory and starts all containers required to run Ente.  + + ## Queries? -If you need any help or support, do not hesitate to drop your queries on our community -[discord channel](https://discord.gg/z2YVKkycX3) or create a -[Github Discussion](https://github.com/ente-io/ente/discussions) where 100s of self-hosters help each other. \ No newline at end of file +If you need support, please ask on our community +[Discord](https://ente.io/discord) or start a discussion on +[GitHub](https://github.com/ente-io/ente/discussions/). diff --git a/docs/docs/self-hosting/museum.md b/docs/docs/self-hosting/museum.md index b55f098a20..2393709e52 100644 --- a/docs/docs/self-hosting/museum.md +++ b/docs/docs/self-hosting/museum.md @@ -1,41 +1,49 @@ --- -title: Building your Museum.yaml +title: Configuring your server description: Guide to writing a museum.yaml --- -## Configuring `museum.yaml` +# Configuring your server -`Museum.yaml` is a YAML configuration file used to configure various things for museum. -By default, [`local.yaml`](https://github.com/ente-io/ente/tree/main/server/configurations/local.yaml) -is also available, but it is overridden if `museum.yaml` file is found. We highly -recommend creating and building your own `museum.yaml` instead of editing `configurations/local.yaml`. -The `my-ente` directory will include a `museum.yaml` file with some configurations around encryption -keys and secrets, postgres DB, and MinIO. +Ente's monolithic server is called **museum**. + +`museum.yaml` is a YAML configuration file used to configure museum. By default, +[`local.yaml`](https://github.com/ente-io/ente/tree/main/server/configurations/local.yaml) +is provided, but its settings are overridden with those from `museum.yaml`. + +If you used our quickstart script, your `my-ente` directory will include a +`museum.yaml` file with preset configurations for encryption keys, secrets, +PostgreSQL and MinIO. > [!TIP] -> Always do `docker compose down` inside `my-ente` directory, if you've made any changes to `museum.yaml` -> and then restart the containers with `docker compose up -d ` to see the changes in action. +> +> Always do `docker compose down` inside your `my-ente` directory. If you've +> made changes to `museum.yaml`, restart the containers with `docker compose up +> -d ` to see your changes in action. -### S3 Buckets +## S3 buckets -By default, the `s3` section is configured to use local minIO buckets and for the same reason -`are_local_buckets` is set to `true`. If you wish to bring any external S3 provider, -you just have to edit the configuration with appropriate credentails and details given by the provider. -And set `are_local_buckets` to false. Check out [Configuring S3](/self-hosting/guides/configuring-s3.md) -to understand more on how to configure S3 buckets and how the communication happens. +The `s3` section within `museum.yaml` is by default configured to use local +MinIO buckets. -MinIO makes use of the port `3200` for API Endpoints and the Client Web App is run over `:3201` -(both on localhost). You can login to MinIO Console Web UI by accessing `localhost:3201` in your web-browser -and setting up all the things related to regions there itself. +If you wish to use an external S3 provider, you can edit the configuration with +your provider's credentials, and set `are_local_buckets` to `false`. -If you face any issues related to uploads then checkout -[Troubleshooting Bucket CORS](/self-hosting/troubleshooting/bucket-cors) and -[Frequently Answered Error related to S3](/self-hosting/guides/configuring-s3#fae-frequently-answered-errors) +Check out [Configuring S3](/self-hosting/guides/configuring-s3.md) to understand +more about configuring S3 buckets. -### App Endpoints +MinIO uses the port `3200` for API Endpoints and their web app runs over +`:3201`. You can login to MinIO Web Console by opening `localhost:3201` in your browser. -Ente Photos Web app is divided into multiple sub-apps like albums, cast, auth, etc. -These endpoints are configurable in the museum.yaml under the `apps.*` section. +If you face any issues related to uploads then checkout [Troubleshooting bucket +CORS](/self-hosting/troubleshooting/bucket-cors) and [Frequently encountered S3 +errors](/self-hosting/guides/configuring-s3#frequently-encountered-errors). + +## App endpoints + +Ente Photos Web app is divided into multiple sub-apps like albums, cast, auth, +etc. These endpoints are configurable in the museum.yaml under the `apps.*` +section. For example, @@ -47,14 +55,15 @@ apps: family: family.myente.xyz ``` -By default, all the values redirect to our publicly hosted production services. -After you are done with filling the values, restart museum and the App will start utilizing -those endpoints for everything instead of the Ente's prod instances. +By default, all the values redirect to our publicly hosted production services. +After you are done with filling the values, restart museum and the app will +start utilizing those endpoints instead of Ente's production instances. -Once you configure all the necessary endpoints, `cd` into `my-ente` and stop all the docker -containers with `docker compose down` to completely stop all the containers and restart them -with `docker compose up -d`. +Once you have configured all the necessary endpoints, `cd` into `my-ente` and +stop all the Docker containers with `docker compose down` and restart them with +`docker compose up -d`. -Similarly, you can read the default [`local.yaml`](https://github.com/ente-io/ente/tree/main/server/configurations/local.yaml) -and build a functioning `museum.yaml` for many other functionalities like SMTP, Discord -Notifications, Hardcoded-OTT's, etc. \ No newline at end of file +Similarly, you can use the default +[`local.yaml`](https://github.com/ente-io/ente/tree/main/server/configurations/local.yaml) +as a reference for building a functioning `museum.yaml` for many other +functionalities like SMTP, Discord notifications, Hardcoded-OTTs, etc. diff --git a/docs/docs/self-hosting/reverse-proxy.md b/docs/docs/self-hosting/reverse-proxy.md index c9b8cf28d3..7ef79ac412 100644 --- a/docs/docs/self-hosting/reverse-proxy.md +++ b/docs/docs/self-hosting/reverse-proxy.md @@ -3,35 +3,34 @@ Title: Configuring Reverse Proxy Description: configuring reverse proxy for Museum and other endpoints --- -## Reverse Proxy +# Reverse proxy -This step isn't really the direct next step after creating an account. It is -one of the most essential steps to avoid certain CORS errors and will help you through -the configuration coming ahead. +Ente's server (museum) runs on port `:8080`, web app on `:3000` and the other +apps from ports `3001-3004`. -Museum runs on port `:8080`, Ente Photos web app runs on `:3000` and so on the other apps -are lined up after each other from ports `3001-3004`. +We highly recommend using HTTPS for Museum (`8080`). For security reasons museum +will not accept incoming HTTP traffic. -We highly recommend using HTTPS for Museum (`8080`). Primarily, because for security reasons Museum -won't accept any incoming HTTP traffic. Hence, all the requests will fail. - -Head over to your DNS Management Dashboard and setup the appropriate records for the endpoints. -Mostly, `A` or `AAAA` records targeting towards your server's IP address should be sufficient. The rest of the work -will be done by the web server sitting on your server machine. +Head over to your DNS management dashboard and setup the appropriate records for +the endpoints. Mostly, `A` or `AAAA` records targeting towards your server's IP +address should be sufficient. The rest of the work will be done by the web +server on your machine.  -### With Caddy +### Caddy -Setting up a reverse proxy with Caddy is pretty easy and straightforward. Firstly, install Caddy -on your server machine. +Setting up a reverse proxy with Caddy is easy and straightforward. + +Firstly, install Caddy on your server. ```sh sudo apt install caddy ``` -After the installation is complete, a `Caddyfile` is created on the path `/etc/caddy/`. This file is -used to configure reverse proxies and a whole lot of different things. +After the installation is complete, a `Caddyfile` is created on the path +`/etc/caddy/`. This file is used to configure reverse proxies among other +things. ```yaml # Caddyfile - myente.xyz is just an example. @@ -44,5 +43,7 @@ ente.myente.xyz { #...and so on for other endpoints ``` -After a few hard-reloads, Ente Photos web app should be up on https://ente.myente.xyz. You can check out -the documentation for any other reverse proxy tool (like nginx) you want to use. \ No newline at end of file +After a hard-reload, the Ente Photos web app should be up on https://ente.myente.xyz. + +If you are using a different tool for reverse proxy (like nginx), please check +out their documentation.