There are multiple ways to extend RubyMine with custom functionality. The first one is writing plugins in Java. Since RubyMine is based on the IntelliJ platform, it supports a large subset of the IntelliJ plugin API (essentially, everything which is not coupled to Java). You can get more information on writing plugins at the PluginDevelopment page.
The second way, specific to RubyMine, is to write extension scripts in Ruby. The extension scripts are run under JRuby and have complete access to the underlying Java API, and in addition to that there's a number of Ruby wrappers around the API that make it easier to write extension scripts in idiomatic Ruby.
RubyMine includes a UI for managing extension scripts: Settings | Extensions. In this dialog, you can see that there's a number of extension scripts shipped with RubyMine, which can be used as examples of using the extension API. You can also specify the location of folders from which your own extension scripts will be loaded.
The extensions API consists of the following main areas:
- Editor actions API
- [Intention actions API]
- [Paramdefs API] (semantic description of function parameters)
- [Dynamic type description API]