# `Membrane.Smelter`
[🔗](https://github.com/membraneframework/membrane_smelter_plugin/blob/v0.12.2/lib/membrane/smelter/smelter.ex#L1)

Membrane SDK for [Smelter](https://github.com/software-mansion/smelter).

## Input streams

Each input pad has a format `Pad.ref(:video_input, input_id)` or `Pad.ref(:audio_input, input_id)`,
where `input_id` is a string. `input_id` needs to be unique for all input pads, in particular
you can't have audio and video input pads with the same id.

See `Membrane.Smelter.Lifecycle` for input stream lifecycle notifications.

## Output streams

Each output pad has a format `Pad.ref(:video_output, output_id)` or `Pad.ref(:audio_output, output_id)`,
where `output_id` is a string. `output_id` needs to be unique for all output pads, in particular
you can't have audio and video output pads with the same id.

After registering and linking an output stream Smelter will notify the parent with
[`output_registered/0`](`t:Membrane.Smelter.Lifecycle.output_registered/0`).

## Composition specification - `video`

To specify what Smelter should render you can:
- Define `initial` option when connecting `:video_output` pad.
- Send [`Request.UpdateVideoOutput`](`Membrane.Smelter.Request.UpdateVideoOutput`)
notification to update a scene on an already connected pad.

For example, code snippet bellow will update content of a stream from `Pad.ref(:video_output, "output_0")`
to include streams from `Pad.ref(:video_input, "input_0")` and `Pad.ref(:video_input, "input_1")`
side by side using [`Tiles`](https://smelter.dev/http-api/components/tiles) component.

```
scene_update_request =  %Request.UpdateVideoOutput{
  output_id: "output_0"
  root: %{
    type: :tiles,
    children: [
      { type: "input_stream", input_id: "input_0" },
      { type: "input_stream", input_id: "input_1" }
    ]
  }
}

{[notify_child: {:smelter, scene_update_request}]}
```

`:root` option specifies root component of a scene that will be rendered on the output stream.
Concept of a component is explained [here](https://smelter.dev/http-api/overview#components). For
actual component definitions see Smelter documentation e.g.
[`View`](https://smelter.dev/http-api/components/view),
[`Tiles`](https://smelter.dev/http-api/components/tiles), ...

## Composition specification - `audio`

- Define `initial` option when connecting `:audio_output` pad.
- Send [`Request.UpdateAudioOutput`](`Membrane.Smelter.Request.UpdateAudioOutput`)
notification to audio composition on an already connected pad.

For example, code snippet bellow will update content of a stream from `Pad.ref(:video_output, "output_0")`
to include streams from `Pad.ref(:video_input, "input_0")` and `Pad.ref(:video_input, "input_1")`
mixed together, where `"input_0"` is mixed at half volume.

```
audio_update_request =  %Request.UpdateAudioOutput{
  output_id: "output_0"
  inputs: [
    { input_id: "input_0", volume: 0.5 },
    { input_id: "input_1" }
  ]
}

{[notify_child: {:smelter, audio_update_request}]}
```

## Notifications

Smelter bin can send following notifications to the parent process.
- [`Lifecycle.notification/0`](`t:Membrane.Smelter.Lifecycle.notification/0`) -
Notification about lifecycle of input/output streams.
- [`Request.result/0`](`t:Membrane.Smelter.Request.result/0`) - Result of a
`Membrane.Smelter.Request` sent from the parent process.

## Smelter documentation

This documentation covers mostly Elixr/Membrane specific API. For platform/language independent topics
that are not covered here check [Smelter documentation](https://smelter.dev/http-api/overview).

## Bin options

Passed via struct `t:Membrane.Smelter.t/0`

- `framerate`  

  ```
  Membrane.RawVideo.framerate()
  ```
  
  ***Required***  
  Framerate of Smelter outputs.

- `output_sample_rate`  

  ```
  output_sample_rate()
  ```
  
  Default value: `48000`  
  Sample rate of audio on Smelter outputs.

- `api_port`  

  ```
  :inet.port_number() | port_range()
  ```
  
  Default value: `8081`  
  Port number or port range where Smelter will be hosted.

- `stream_fallback_timeout`  

  ```
  Membrane.Time.t()
  ```
  
  Default value: `500000000`  
  Timeout that defines when Smelter should switch to fallback on
  the input stream that stopped sending frames.

- `composing_strategy`  

  ```
  :real_time_auto_init | :real_time | :offline_processing
  ```
  
  Default value: `:real_time_auto_init`  
  Specifies Smelter mode for composing frames:
  - `:real_time` - Frames are produced at a rate dictated by real time clock. The parent
  process has to send `:start_composing` message to start.
  - `:real_time_auto_init` - The same as `:real_time`, but the pipeline starts
  automatically and sending `:start_composing` message is not necessary.
  - `:offline_processing`
    - Output streams will be produced faster than in real time if input streams are
    ready.
    - Never drop output frames, even if the encoder or rendering process is not able to
    process data in real time.
  
      When using this option, make sure to register the output stream before starting;
      otherwise, Smelter will run in a busy loop processing data far into the future.

- `server_setup`  

  ```
  :already_started
  | {:already_started, :inet.ip_address() | String.t() | atom()}
  | :start_locally
  | {:start_locally, path :: String.t()}
  ```
  
  Default value: `:start_locally`  
  Defines how Smelter bin should start-up a its server.
  
  Available options:
  - `:start_locally` - Smelter server is automatically started.
  - `{:start_locally, path}` - Smelter server is automatically started, but different binary
  is used to spawn the process.
  - `:already_started` - Smelter bin assumes, that the server is already started
  and is available on a localhost on a specified port. When this option is selected, the
  `api_port` option need to specify an exact port number (not a range).
  - `{:already_started, ip_or_hostname}` - Smelter bin assumes, that the server is already
  started and is available on a specified IP or name on a specified port. When this option is
  selected, the `api_port` option need to specify an exact port number (not a range). If
  `ip_or_hostname` is a string or an atom, it will be resolved to `:inet.ip_address()` using
  `:inet.gethostbyname/1`.

- `init_requests`  

  ```
  list(Request.t())
  ```
  
  Default value: `[]`  
  Requests that will be sent on startup to Smelter server. It's main use case is to
  register renderers that will be needed in the scene from the very beginning.
  
  Example:
  ```
  [%Request.RegisterShader{
    shader_id: "example_shader_1",
    source: "<shader sources>"
  }]
  ```

## Pads

### `:audio_input`

Accepted formats:
```
%Opus{self_delimiting?: false}
```
```
%RemoteStream{type: :packetized, content_format: Opus}
```
```
%RemoteStream{type: :packetized, content_format: nil}
```

Direction: | `:input`
Availability: | `:on_request`
Pad options:

- `required`  

  ```
  boolean()
  ```
  
  Default value: `false`  
  If stream is marked required Smelter will delay processing new frames until
  frames are available. In particular, if there is at least one required input stream and the
  encoder is not able to produce frames on time, the output stream will also be delayed. This
  delay will happen regardless of whether required input stream was on time or not.

- `offset`  

  ```
  Membrane.Time.t() | nil
  ```
  
  Default value: `nil`  
  An optional offset used for stream synchronization. This value represents how PTS values of the
  stream are shifted relative to the start request. If not defined streams are synchronized
  based on the delivery times of initial frames.

- `port`  

  ```
  :inet.port_number() | port_range()
  ```
  
  Default value: `{10000, 60000}`  
  Port number or port range.
  
  Internally Smelter server communicates with this pipeline locally over RTP.
  This value defines which TCP ports will be used.

### `:video_input`

Accepted formats:
```
%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
```

Direction: | `:input`
Availability: | `:on_request`
Pad options:

- `required`  

  ```
  boolean()
  ```
  
  Default value: `false`  
  If stream is marked required Smelter will delay processing new frames until
  frames are available. In particular, if there is at least one required input stream and the
  encoder is not able to produce frames on time, the output stream will also be delayed. This
  delay will happen regardless of whether required input stream was on time or not.

- `offset`  

  ```
  Membrane.Time.t() | nil
  ```
  
  Default value: `nil`  
  An optional offset used for stream synchronization. This value represents how PTS values of the
  stream are shifted relative to the start request. If not defined streams are synchronized
  based on the delivery times of initial frames.

- `port`  

  ```
  :inet.port_number() | port_range()
  ```
  
  Default value: `{10000, 60000}`  
  Port number or port range.
  
  Internally Smelter server communicates with this pipeline locally over RTP.
  This value defines which TCP ports will be used.

### `:audio_output`

Accepted formats:
```
%RemoteStream{type: :packetized, content_format: Opus}
```

Direction: | `:output`
Availability: | `:on_request`
Pad options:

- `port`  

  ```
  :inet.port_number() | port_range()
  ```
  
  Default value: `{10000, 60000}`  
  Port number or port range.
  
  Internally Smelter server communicates with this pipeline locally over RTP.
  This value defines which TCP ports will be used.

- `encoder`  

  ```
  Encoder.Opus.t()
  ```
  
  ***Required***  
  

- `channels`  

  ```
  :stereo | :mono
  ```
  
  Default value: `:stereo`  
  

- `send_eos_when`  

  ```
  send_eos_condition()
  ```
  
  Default value: `nil`  
  Condition for automatically finishing output stream in response to end of input streams.
  
  - `{:any_of, input_ids}` - End the output stream if any of the inputs from the list finished
  or if they don't exist.
  - `{:all_of, input_ids}` - End the output stream if all of the inputs from the list finished
  or if they don't exist.
  - `:any_input` - End the output stream when any input stream finishes.
  - `:all_inputs` - End the output stream when all of the input streams have finished. This
  also includes a case where no inputs are were ever connected.

- `initial`  

  ```
  any()
  ```
  
  ***Required***  
  Initial audio mixer configuration that will be produced on this output.
  
  Example:
  ```
  %{
    inputs: [
      %{ input_id: "input_0" },
      %{ input_id: "input_0", volume: 0.5 }
    ]
  }
  ```
  
  To change the scene after the registration you can send `%Request.UpdateAudioOutput{}`
  notification.

### `:video_output`

Accepted formats:
```
%Membrane.H264{alignment: :nalu, stream_structure: :annexb}
```

Direction: | `:output`
Availability: | `:on_request`
Pad options:

- `port`  

  ```
  :inet.port_number() | port_range()
  ```
  
  Default value: `{10000, 60000}`  
  Port number or port range.
  
  Internally Smelter server communicates with this pipeline locally over RTP.
  This value defines which TCP ports will be used.

- `width`  

  ```
  non_neg_integer()
  ```
  
  ***Required***  
  

- `height`  

  ```
  non_neg_integer()
  ```
  
  ***Required***  
  

- `encoder`  

  ```
  Encoder.FFmpegH264.t()
  ```
  
  ***Required***  
  

- `send_eos_when`  

  ```
  send_eos_condition()
  ```
  
  Default value: `nil`  
  Condition for automatically finishing output stream in response to end of input streams.
  
  - `{:any_of, input_ids}` - End the output stream if any of the inputs from the list finished
  or if they don't exist.
  - `{:all_of, input_ids}` - End the output stream if all of the inputs from the list finished
  or if they don't exist.
  - `:any_input` - End the output stream when any input stream finishes.
  - `:all_inputs` - End the output stream when all of the input streams have finished. This
  also includes a case where no inputs are were ever connected.

- `initial`  

  ```
  any()
  ```
  
  ***Required***  
  Initial scene that will be rendered on this output.
  
  Example:
  ```
  %{
    root: %{
      type: :view,
      children: [
        %{ type: :input_stream, input_id: "input_0" }
      ]
    }
  }
  ```
  
  To change the scene after the registration you can send `%Request.UpdateVideoOutput{}`
  notification.
  
  Format of the `:root` field is documented [here](https://smelter.dev/http-api/overview#components).
  For specific options see documentation pages for each component e.g.
  [`View`](https://smelter.dev/http-api/components/view),
  [`Tiles`](https://smelter.dev/http-api/components/tiles), ...

# `audio_input_pad_opts`

```elixir
@type audio_input_pad_opts() :: [
  required: boolean(),
  offset: Membrane.Time.t() | nil,
  port: :inet.port_number() | port_range()
]
```

Options for pad `:audio_input`

# `audio_output_pad_opts`

```elixir
@type audio_output_pad_opts() :: [
  port: :inet.port_number() | port_range(),
  encoder: Membrane.Smelter.Encoder.Opus.t(),
  channels: :stereo | :mono,
  send_eos_when: send_eos_condition(),
  initial: any()
]
```

Options for pad `:audio_output`

# `input_id`

```elixir
@type input_id() :: String.t()
```

Input stream id, uniquely identifies an input pad.

# `output_id`

```elixir
@type output_id() :: String.t()
```

Output stream id, uniquely identifies an output pad.

# `output_sample_rate`

```elixir
@type output_sample_rate() :: 8000 | 12000 | 16000 | 24000 | 48000
```

Supported output sample rates.

# `port_range`

```elixir
@type port_range() ::
  {lower_bound :: :inet.port_number(), upper_bound :: :inet.port_number()}
```

Range of ports.

# `send_eos_condition`

```elixir
@type send_eos_condition() ::
  nil
  | :any_input
  | :all_inputs
  | {:any_of, [input_id()]}
  | {:all_of, [input_id()]}
```

Condition that defines when output stream should end depending on the
EOS received on inputs.

# `t`

```elixir
@type t() :: %Membrane.Smelter{
  api_port: :inet.port_number() | port_range(),
  composing_strategy: :real_time_auto_init | :real_time | :offline_processing,
  framerate: Membrane.RawVideo.framerate(),
  init_requests: [Membrane.Smelter.Request.t()],
  output_sample_rate: output_sample_rate(),
  server_setup:
    :already_started
    | {:already_started, :inet.ip_address() | String.t() | atom()}
    | :start_locally
    | {:start_locally, path :: String.t()},
  stream_fallback_timeout: Membrane.Time.t()
}
```

Struct containing options for `Membrane.Smelter`

# `video_input_pad_opts`

```elixir
@type video_input_pad_opts() :: [
  required: boolean(),
  offset: Membrane.Time.t() | nil,
  port: :inet.port_number() | port_range()
]
```

Options for pad `:video_input`

# `video_output_pad_opts`

```elixir
@type video_output_pad_opts() :: [
  port: :inet.port_number() | port_range(),
  width: non_neg_integer(),
  height: non_neg_integer(),
  encoder: Membrane.Smelter.Encoder.FFmpegH264.t(),
  send_eos_when: send_eos_condition(),
  initial: any()
]
```

Options for pad `:video_output`

# `options`

```elixir
@spec options() :: keyword()
```

Returns description of options available for this module

---

*Consult [api-reference.md](api-reference.md) for complete listing*