Do You Want to Build a Server?

To the tune of “Do You Want to Build a Snowman” by Kristen Bell

Junior Bro:
Bro?
Do you want to build a server?
Come on let’s go and code
I never see your push commits
Or your tickets,
It’s like you’ve gone away
We used to be code buddies
And now we’re not
I wish you would tell me why
Do you want to build a server?
It doesn’t have to be a server

Senior Brotato: GTFO bro

Junior Bro: kthx bye

Junior Bro:
Do you want to build a server
Or push commits at the pub?
I think some tickets are way overdue
I’ve started talking to
HR lad down the halls
(Hang in there, bro!)
It gets a little lonely
All these empty chatrooms
Just watching the hours tick by
(Tick-tock tick-tock tick-tock tick-tock)

Junior Bro:
Bro?
Please I know you’re in there,
People are asking where you’ve been,
They say, “have a NOS”;
And I’m trying to,
I’m right out here for you
Just let me in
We only have each other
It’s just you and me
What are we gonna do?
Do you want to build a server? (sniff)

Firefox Integrating Pocket

Firefox had a nifty feature: “Reading List.” And then it didn’t. Now it has a proprietary service in its place: Pocket. Instead of being able to locally save things to read later, you now have to find a plugin or use their integrated service. We have updated our previous post to reflect these changes. Pocket integration, just like Firefox Hello, should be an optional plugin, not integrated into the core of the browser. Mozilla could have easily promoted Wallabag instead of a proprietary “freemium” service.

Mozilla is reducing the “free and open source” nature of Firefox, which is disconcerting.

Fixing Firefox and Enhancing Privacy

Between Mozilla adding more “features” to Firefox, and malicious scripts on websites, we have been recommending the following changes and add-ons to Firefox users.

Please note that these are only suggestions; if you disagree with something, you don’t have to do it (but we would love to hear why in the comments)! We are not going to go into detail about how great each plugin is – we expect that you will exercise due diligence before using anything described in this post.


New Tabs

To get rid of the “recommendations” and ads inside Firefox new tabs, click the gear on the top right, and select “Classic” or “blank.”


about:config

Unless you have visited this before, you will see a prompt with a statement that you should be careful when editing anything.

Firefox Hello
Search for loop.enabled and set to “false” by double-clicking.

Reason: Firefox Hello uses Tokbox, which is owned by Telefónica Digital, a subsidiary of Telefónica (a Spanish broadband and telecommunications provider). We do not appreciate an ISP service being integrated into the web browser. This should be a plugin, not a core feature.

Pocket
Added June 2, 2015
Search for browser.pocket.enabled and set to “false” by double-clicking.

Reason: Pocket is a proprietary “save it for later” page reader that has replaced Firefox’s “Reading List” feature. It is yet another closed-source integration that should have been a plugin. It requires a 3rd-party login (including Firefox account). An open source alternative to Pocket is Wallabag.

Google Safe Browsing
Search for browser.safebrowsing.*enabled and set all to false by double-clicking.

Note: Some of this can also be turned off by navigating to Menu -> Options -> Security (or clicking here) and ensuring “Block reported web forgeries” and “Block reported attack sites” are un-checked.

Reason: This is the “Google Safe browsing” extension. While many might find this useful, we do not like the pages we visit being sent back to Google. Turning this off can be perceived as making your browser less secure, so make sure that you are careful what links you click. uBlock, NoScript and Ghostery pick up some of the slack in helping prevent malicious attacks.


Add-ons

Each Scoundrel has their own preference in which add-ons they use. Here is the list of add-ons that we all agree are useful.

HTTPS_Everywhere_new_logoHTTPS Everywhere – “HTTPS Everywhere is a Firefox, Chrome, and Opera extension that encrypts your communications with many major websites, making your browsing more secure. ”

noscript NoScript Security Suite – “Allow active content to run only from sites you trust, and protect yourself against XSS and Clickjacking attacks.”

ublockuBlock – “An efficient blocker: easy on memory and CPU footprint, and yet can load and enforce thousands more filters than other popular blockers out there.”

ghosteryGhostery – “See who’s tracking your web browsing and block them with Ghostery.”

 

uBlock Note: Make sure to go to Menu -> Add-ons and click on uBlock Options, and scroll down and click Show Dashboard to customize. Alternatively, you can visit this link.

Migrating from GitHub to our GitLab

We have started to migrate our project development away from GitHub and on to our own self-hosted GitLab. All current public project code that is available on that website will still be there, and we are not completely removing GitHub from our lives. The current plan of action is to have our main repository be on our own server, which will push updates to GitHub once a day (probably around 11PM PST).

The Scoundrels have always been active in the Free and Open Source Software communities, and we are always finding more ways to supplant proprietary tools with F/OSS ones. It seemed rather odd, and quite obvious, that we needed to remedy something quite crucial: we have been relying on an external service for virtually all of our code hosting needs.

GitHub Issues and Wiki pages will be closed down as we migrate to our own server. We aim to push code to GitHub as a sort of “middle layer” between us and those who do not want to sign up on our server. We will probably accept Pull Requests through GitHub (not guaranteed), but issue management will be through GitLab.

Our GitLab uses a certificate signed by us, so your browser will most likely give you a warning. As stated on our Trust page, only accept the security exception (or install our root certificate) if you trust our services and only if you plan on developing within our GitLab.

Planning for Scrib 2.3.0

Scribbington

With the ability for scrib plugins to register as “Listeners,” I have been re-thinking how core pieces are defined. With the amount of waffling1Seriously, look at scrib-inactive’s commit log. Gross. that I have done in the past in regard to how I want to store brain-data for Scribbington and BaconCat, I have realized that I should make the brain and Markov generator as plugins. This would greatly expand the ability to change how the brain works (and even switch out technologies) without having to constantly muck up the core code.

Essentially, scrib, at its core, is becoming a simple messenger bot. It accepts input and passes everything to Listener Plugins, with commands being sent to Action Plugins.

Right now each plugin is a simple .py file inside the /plugins/ folder. With the new system, I would like to have it dig one folder-depth deeper, so that each plugin is in its own area. This will allow for easier plugin management2Not to mention, we can then simply “git clone” plugins! and help register namespaces3I recently had an issue where two plugins used the same command; both executed, which was confusing., which needs to be done. Also, if a plugin registers as a Listener and an Action, the PluginManager loads the module twice. This grosses me out4You should be grossed out to. Let’s be grossed out together!.

I want plugins to be able to register dependencies on other plugins, too. For example, the brain currently in scrib requires Redis. I would love to have the Redis-based code in its own plugin so that I can use it with other plugins (and not duplicate code).

With this plugin idea5If I explained well enough what I want to do with plugins, that is., virtually everything else that we want to do with scrib becomes a plugin. I think that overall, this will make scrib far more manageable and have less krufty commits.

It has been fun writing plugins for scrib; now I want to make it easier for others. Scribbington, BaconCat, and various other bots will be able to further diverge from each other while maintaining the same basic core, which (to me) is a plus. The more we can customize bots to our needs, the better.

In case you are curious, the various plugins I have been toying with are in their own repository (GitHub mirror). They have mostly been me playing around with APIs and trying to break the scrib messaging system6I must say, it needs to be rewritten eventually.. The only plugin that actually does something other than API calls is Sociocide, which is going to be a fun chatkill game. I just need to build a registration system that plugins can use, instead of the wonky-and-easily-spoofed one that’s baked in. That plugin is actually the catalyst for everything that has led up to this post.

Completely off-topic: I am so glad I decided to go with footnotes. It makes everything so much easier to read.

   [ + ]

1. Seriously, look at scrib-inactive’s commit log. Gross.
2. Not to mention, we can then simply “git clone” plugins!
3. I recently had an issue where two plugins used the same command; both executed, which was confusing.
4. You should be grossed out to. Let’s be grossed out together!
5. If I explained well enough what I want to do with plugins, that is.
6. I must say, it needs to be rewritten eventually.

Code Cleanup: Scoundrels 0.4.6

There is a Python library that I have built for The League of Magnificent Scoundrels that is a (small) collection of code that we use repeatedly. It’s only taken me forever, but I finally implemented SSL into the IRC connector portion… which means that scrib now can connect to SSL/TLS encrypted chats.

After I do a better job at loading interfaces, and there is a REPL for scrib1We want to send commands (including debug/maintenance ones) without having to do it in a chat box/restart the bot., I will most likely make it required. So, I guess this is me saying that the Scoundrels IRC library is going to be encryption-only in the near future. If you use it, consider this your [only] heads up.

Another thing that I am going to do with the Scoundrels Python library is finally clean up barf. There’s a lot of legacy functions for backwards compatibility, but every still-active project has already moved to using the more uniform commands2This reminds me, scoundrels.cfg still uses the old style, and needs a general update…:

Finally, scoundrels.logger really needs to be overhauled. I forgot to set scrib’s log path3Setting a program’s log path should be required without a fallback the way it is handled now. and only found out that ~/.scoundrels/scrib.log was 18.8mb from doing some dotfile housecleaning.

The next version. 0.4.6, will include updated tests, as well. Those things need to be revisited.

I’m just glad it was not hard to upgrade these tools to Python 3.x.

   [ + ]

1. We want to send commands (including debug/maintenance ones) without having to do it in a chat box/restart the bot.
2. This reminds me, scoundrels.cfg still uses the old style, and needs a general update…
3. Setting a program’s log path should be required without a fallback the way it is handled now.

Securing Shell Access via SSH

Goal: Mitigate and securing shell access via ssh from:

  • SSH brute force attacks
  • SSH dictionary attacks
  • Buffer overflow attack

This assumes your sshd_config is located at /etc/ssh/sshd_config. In your sshd_config and that you have sudo and nano.

Open sshd_config for editing: sudo nano /etc/ssh/sshd_config

  • Change Port 22 to something unused on your server.
    • Update iptables/ufw/firewall to deny 22 and allow new port.
  • Bind to a single IP address: ListenAddress 1.2.3.4
  • Ensure Protocol 2 is used.
  • Set PermitRootLogin to no
  • Set DenyUsers root

Optionally, you can set a banner (Banner /path/to/bannertext). This is useful if you want to show a legal disclaimer regarding login attempts:

Now restart the ssh daemon:

  • Debian-based: sudo service ssh restart
  • RedHat-based: sudo service sshd restart
  • Arch-based: sudo systemctl restart sshd.service

Disable SSH Password Login

The best way you can secure from login attempts is to use a key pair login. This portion assumes you do not have an ssh public/private key pair. If you do, skip the generation step. Please note that <port> should be the port you chose above.

  • On a local terminal, issue ssh-keygen -t rsa and follow the prompts. Accept the highest key instead of the default 2048.
  • When that is finished, issue ssh-copy-id -i ~/.ssh/id_rsa.pub user@host to send the server your key1Thank you, Tiger!P, for commenting about this..

Now we can ssh -p <port> user@host into your server. Upon sucess, you can edit your sshd_config and set PasswordAuthentication no and then restart the ssh daemon.

Since the ssh key pair handles authentication, we do not need to have a password login for ssh. Depending on how you set up your ssh key, you may still be prompted for a password. This is the key pair password, not your ssh user.

Please understand that you must back up your local ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub or you will lose access to your server. If you have multiple computers that connect to your server, I suggest you generate keys for each and add them to your ~/.ssh/authorized_keys2 on the server.

   [ + ]

1. Thank you, Tiger!P, for commenting about this.

Roadmap for scrib 2.2.0

Some ideas that have been floating in my head after talking with people about scrib, mainly because I have repeatedly been asking myself the question, “what do I want scrib to do?” I know what I want it to be, I am just not sure what order to build in features. As it stands now, I think that the order is going to be:

  • Brain Management System (diagnostics console?)
  • Personality Index
  • Trainability
  • FAQ/Trivia (and other prepared statements)

I see these as having a cascading effect: in order to get started on any forward process, scrib needs a new information storage system. We started with python marshal, then moved to cPickle, json, and then I did a crazy and killed everything just to use Redis. We are most definitely keeping json as a core tool, because it allows us to transport data to other programs that don’t speak in Python-centric formats.

Brain Management System

When I last talked with KittyTristy about this, we decided that the logging system will be the base from which scrib learns (by default). At a given time interval, scrib will process the logs that it has not processed yet. The processing will most likely include some form of NLTK tagging so that there is a better context for reply generation. The processed information will be stuffed into a MySQL database so that we can maintain relational data as well as do other nifty things outside of the core scrib. Being able to audit and modify the data through some sort of management system will be paramount, as past experience has highlighted.

Personality Index

After we have the brain in working order, I want to have a Personality Index layer, where each instance of the bot can have its own predefined attitude, likes, and other predispositions. BaconCat and version 1.9.9 Scribbington have this, but only because they have divergent brain data. I want to be able to have the bot add and remove things to a list of items it likes, when asked, ‘Do you like bees?’ (If any of you know about Scribbington, you will know he obsesses over his “Three B’s”: boobs, butts, and bees.) BaconCat, likes bacon and DragonLance in inordinate amounts. Both are known to have attempted writing their own flavors of erotica. Be glad you haven’t seen it.

Trainability

Now that they would have their own personality parameters, we want to be able to actually train them. Right now, any and all text that goes through processing gets added. They just homogenize it all, bad and good. Having guidelines for “acceptable” material, allowing specified people to [un]teach, ignore people, or other administrative tasks, would be amazing. This would directly tie into both the Personality Index and Brain Management System.

FAQ/Trivia

One of the most requested features for scrib is the ability to handle predefined dictionaries of questions, be it a FAQ, trivia, or quotes. There is at least one case where this would also include forum and chat rules, so that there can be more automated moderation (when you need to be all ages, you really need to make sure topics stay as such).

With scrib 2.1.0 nearly finished, I am glad to finally have figured out where we are going to go forward.

Is there something you would like to see scrib do, or handle? Send myself (@cptmashek) or @thescoundrels a tweet!

Thanks for reading!

Scrib 2.0.7 Released

Hello!

I just recently coded, tested, and released scrib 2.0.7! I’m excited where this is going, and am also realizing that the departure from the old code has been a mixed blessing. But hey, that’s how learning goes, right?

So, here’s what’s new:

  • Configuration files have been moved to a .scrib/ folder in your home directory (C:Users%user%.scrib in Windows, ~/.scrib in Linux).
  • Logs: There is now a cenral location within .scrib/bots/scrib/logs/ where log files are stored. Currently, all logs are lumped together in that one place, regardless of which bot you might be running.
  • BeautifulSoup4 has been added to help clean up pesky string types. In the pre-2.0 scrib, the brain was often crunchy and awful because of mixed ASCII and Unicode. Gross.
  • One-Line chatting: It is now possible to (in a command line) execute scrib -t [name] "Some sentence you want to speak to scrib." It isn’t terribly pretty right now, because that isn’t the point. Being able to interact with scrib this way enables a lot of programs outside of the Python realm to utilize its features. Of course, this is in no way meaning that we will not be making an API. We just aren’t focusing on that. Yet.
  • make.bat: For some developers (myself included), nothing is worse than being stripped of your ability to continue coding, regardless of what OS you’re on. Running tests and making builds should be done no matter where you’re at. (Fun fact: scrib 2.0.7 was coded, tested, and built in Windows 8.1.)

In addition to this, there were some minor cleanups and things, but nothing else terribly exciting.

KittyTristy and I have decided to make sure the core is solid in both scrib as well as roguekit before we add any more features to either.

For scrib, that means I am going to work to close all existing tickets before setting up anything new. After we reach the 2.1.0 milestone, I shall embark on some rather nifty things, like a new brain system and format, an NLTK plugin, a Twitter plugin (as per requests), and the beginnings of an API.

By the way, our family of scrib-based bots has a new home; it is where I have moved the quotes.

Scrib: Logging and Training

It has been a long while since I have openly talked / blogged about scrib. Most of the chatting about it has been done in our IRC channel, and a lot of the code that has been written has been rather rapid and not really well documented. All of this really needs to change, considering we keep writing things, and then deciding that we’re doing it wrong. Then, we end up rewriting it, or moving to different technologies, and the cycle continues.

Where we’re at

As it stands right now, starting from scrib 1.9.9, I have rewritten pretty much all of the code. This started because Tristy had begun developing a graphical interface for scrib. This brought me to the realization that the command line and IRC interfaces were basically standalone applications that used scrib like a library.

That brought me to the conclusion that I wanted to be able to let people use scrib via the Python Package Index (pypi), and begun rewriting everything so that it would work within the packaging architecture. So, right now (version 2.0.6) is available via pip, but unless you download an existing application or start building one, it is rather useless. That said, I have seen some interaction from random people, which makes me giddy and has fueled a fire to adopt a coding practice to shape scrib up to be something more awesome than it is. So, where do we go from here? The first thing I want to tackle is logging and data storage.

Data Storage

The more I have been talking with people, the more I think I am going to implement SQLAlchemy so that we can use a sql backend for standardized data storage. This will enable us to allow sqlite usage on Android devices, MySQL/MariaDB/whatever-else-sql for whatever flavor a user decides to want to use. Ideally, the zero-config version of scrib would default to sqlite so that could run “out of the box.”

The biggest issue I am currently mulling over, is just how to handle log access by external applications. I want to provide a standard way for scrib to manage logs so that applications don’t need to write their own (but not forbid them from doing so). This means that there needs to be a system in place that will collect the same types of data, whether it be command line interaction or IRC / group messaging systems. Right now, this is what I have started collecting:

If you were to build an application using scrib, what other data would you see wanting to log?