Jump to content


Photo
- - - - -

The Nitty Gritty of the Server Move


  • Please log in to reply
5 replies to this topic

#1 Ken White

Ken White

    Foul Owl on the Prowl

  • Administrators
  • 3,238 posts

Posted 19 June 2015 - 04:30 PM

This thread is for those who like to watch hours of behind-the-scenes footage and techwonk heads. It will be my back and forths with subhosting as we go along with the move - what I'm telling them, what they're telling me. So let's begin with my blueprint to them:

Hey Ivan –

This is an overview of what we’re looking to do with the move, the new servers and how they should be configured. Can get into the nitty-gritty specifics of the actual domains, etc. over the next 10 days as we get closer to the move.

TMUnderground.com

TMUnderground.com will be the umbrella over all the other parts. Our preference will be to have subdomains for each part of the overall site – movies.tmunderground.com, forums.tmunderground.com, etc., both for SEO reasons and that it just looks better visually. Immediately we’ll need movies, forums, radio, live, and tmucloud (our rebranding of owncloud). There may be a couple of things in the future, but I’ll do that myself when the time comes.

Here’s a couple of questions – I assume this can all be done with a virtual domain (tmunderground.com) to incorporate sites under the domain (or as subdomains) on both servers? Your guys would have more information about doing it that way. Also what kind of issues would we face taking existing sites (the existing Sharemixer running on tmunderground, as well as our vBulletin forums at http://www.tmoaradio...vb/activity.php - we’ll be switching over to IP Boards in the middle/end of July, and they have good conversion scripts, but it’s all about the links (and I can research moving an existing site to a different domain on my own).

Does Sharemixer care where it’s installed, or are the internal links somehow hard-coded so if it goes into a subdomain, everything breaks? If that’s the case, would it be possible to do a new install of the final revision of Sharemixer, then move the database and data over and recreate it temporarily on the subdomain, at least until we do what we need to do with Viewshark – which we’ll buy mid-July and hopefully deploy sometime in August. The radio site will be rebuilt from scratch, as will the live broadcast site (we’ll be using DaCast - http://www.dacast.com/ - so it won’t have any effect on our site, with all bandwidth, conversion, etc. on their site and only an embed on ours.) Haven’t decided which way we’re going with our chat solution – will either be self-hosted or hosted, depending on the solution I want to go with. One self-hosted one I’m looking at is this: http://www.flashcoms..._chat/overview/ - the only issue I see is that the server would require:

The application requires streaming media server. We support all market available media servers, so you can choose one of them:
· Adobe Flash media server www.adobe.com
· Wowza media server www.wowzamedia.com
· Red 5 www.red5.org

And I don’t know what kind of issues, if any, that might present. In any case, chat, etc. is for August, not the immediate July move, so it’s not really high on my priorities at the moment. So we’ll figure out what will work and what won’t as we get closer to deploying it.

Other sites

I have about a dozen non-tmunderground releated domains. They need to be moved, though only one of them is really active right now (a wordpress site). The rest are for testing or future projects, and can go on either server, since they’re standalone.

Viewshark development

As I mentioned when we chatted, my idea is to install Viewshark in the latter part of July, preferably on a separate IP for development (can talk to them about how that will be affected when it’s moved to movies.tmunderground.com). How many IPs do we have to work with? I normally don’t care or even use the extra IPs I have on a hosting plan, but in this case, since we’ll be loading Viewshark with all the existing Sharemixer data (from video files to database stuff) and then working to get it into shape before deployment, I think just having it accessible by IP will work best – we can work with it, my members can check it out, but it won’t be on a specific domain until it moves to its final home.

There are two aspects to the switch from Sharemixer to Viewshark. First (and probably most important) is the database conversion. My guy looked at an earlier script we were looking at, and they sent us a blank database to get an idea of how it was laid out. He was able to pretty quickly and easily do the conversion, since all of them seem to be pretty similar to Sharemixer’s tables, etc. By midweek next week, I should know if he’s going to have the time to do the conversion to Viewshark. If not, I assume you have SQL guys on staff who can knock it out – it didn’t seem to present much of a challenge. We can work out the cost at that time.

Second aspect is the actual flv files. Every script I’ve looked at doesn’t dump them in one directory, like Sharemixer, but instead breaks them into subdirectories, by user ID. The database does keep track of what videos, audio, etc. belongs to what user, but in the file system, they’re split out. My ideal solution would be to have the file system created automatically, based on the information in the database. What they actually said when I talked to them was:

2. The file system is something like that yes. An actual server path to a video file would look like /home/user/public_html/f_data/data_userfiles/user_media/mRaY93qUpGn8/v/7Va8nRa2iDbQ4441.360p.mp4
After "user_media" comes the [user key] folder, then "v" folder for videos and the file name consists of the database file_key.(360p/480p/720p/mob/).mp4
And the thumbnails are stored in /f_data/data_userfiles/user_media/[user_key]/t/[file_key]/0.jpg, 1.jpg, 2.jpg, 3.jpg where 0.jpg is a larger thumbnail, something like 640x480, and 1.jpg, 2.jpg, 3.jpg and smaller thumbs
Note: our script does not convert to FLV anymore, now it does MP4 only, however if you rename your files correctly to file_key.360p.flv then the files should get recognized for playing.

So we’ll need a file system based on user ID, and then either v (for video), presumably a (for audio), etc., and then finally the actual video files, which include the file name from the database and the resolution. I’m not exactly sure about if the database calls a movie 10352.flv, if renaming it to 10352_480p.flv will work, or if Viewshark actually requires their filename for the video file. More to talk to them about.

I’ve done a little research, and it doesn’t look that difficult for someone who knows PHP to write a script to read a database and create a file system based on specific information (user ID) along with subdirectories like v. Another question I have for them is will the existing user IDs (which are just numbers) be readable by their script rather than the alphanumeric string their script creates – this, of course, is only important for existing accounts – new accounts will use whatever their script generates.

The perhaps-harder thing that I’d like to see is a php script that reads the video files under each account in the database and moves them from the all-in-one Sharemixer directory to the proper directory for the new file system. If it’s not doable, then I can always put in a few days of grind work and move them manually. (which brings up what I mentioned in an earlier email – I’d really like some kind of accessible desktop on the server so I can do the Linux equivalent of RDPing into it. Though I can usually work things out in the shell, there are some tasks that are a whole lot faster with a visual interface.)

I think that’s enough for now – pass this on to your tech guys and let me know what will work, what won’t, and what questions you might need answered, specifically from Viewshark, other than the obvious ones I’ve mentioned.

Thanks.

Ken

TMOAsig.png
TMU_forum01.png


#2 Johnny Ex

Johnny Ex

    Cliffhanger

  • Members
  • 1,736 posts
  • LocationSecret lair

Posted 20 June 2015 - 12:09 AM

Holy shit that is a whole mouthful Pappy.
No Im not the tech savvy youngling nerd that I would secretly hope not to be?
All I know is Im seeing IT HAPPEN and I guess at THIS end thats all that matters, sorta....
You are in fact the proverbial man Ken and Im not the only one who is GRATEFUL.
This is a monumental event, I can appreciate all those numbers cuz I know they must mean something good.
All hail and long live the effin U!
Posted Image

#3 Ken White

Ken White

    Foul Owl on the Prowl

  • Administrators
  • 3,238 posts

Posted 03 July 2015 - 02:54 PM

Okay, the data move has begun, and I'm hashing out the details with them of what goes where and what we need. Here's what I sent them:

Hi Lijeesh -

Okay, as I said, I would like tmunderground.com to be the main domain for many of my current active domains, with those existing domains to be subdomains on tmunderground.com.

So we've got:

tmunderground.com (currently on it's own server and not on the VPSs) - the existing Sharemixer install. I'd like that on a subdomain movies.tmunderground.com.

tmoaradio.com/vb - This is a vBulletin forum (though we'll be converting to IP Boards in the coming weeks). I'd like that on a subdomain forum.tmunderground.com

https://107.181.179.89:2199 currently houses our Centova Cast install (streaming internet radio) - if we can put that Centova Cast install on its own IP, and make that IP the subdomain radio.tmunderground.com, that would be optimal.

I'll need a subdomain live.tmunderground.com - that subdomain will be a simple page, with an embedded live player from Dacast and an embedded chat window from whoever we go with for chat services (I mentioned in the email a chat solution that requires a streaming media server, though Adobe Media Server and Wowza are a bit pricy, and I'm not sure I want to go with Red5, which is Java-based) - I may ultimately go with a hosted chat service that will let me embed it wherever I need it)

tmuniversity.com currently has our Owncloud install, and I'd like that data moved to cloud.tmunderground.com.

That covers the tmunderground subdomains. I'm obviously not an expert, or even very smart when it comes to this stuff, but I envision tmunderground.com to be merely a domain that holds the various subdomains. If someone does go directly to http://www.tmunderground.com, they'll get a landing page (which I may do with IP Boards CMS module) that will link them to whatever they want - movies, forum, live, radio, whatever. Probably as a virtual domain, so you can break the various subdomains out to different servers as needed.

In terms of load, probably movies.tmunderground.com will need the most resources, since it will be encoding uploaded video, either with Sharemixer or when we make the conversion to Viewshark. Forums.tmunderground.com is just going to be forum software.

Radio.tmunderground.com will run Centova, which requires limited resources - you can see the system requirement here: http://www.centova.c...st/requirements - all of the Shoutcast streaming server stuff is installed when I install the Centova package. We do not currently use the autodj function, which does server-side encoding of stored mp3s, though it's not beyond the realm of possibility that we might in the future. However, if the server is powerful enough to encode video (and both of them clearly are), I don't imagine they'll have any trouble encoding audio files.

And as I said, live.tmunderground.com will be a page with embeds from other sources, so no resources are needed, and Owncloud on cloud.tmunderground.com doesn't use much in the way of resources, since it's primarily just storage.

As for non-tmunderground domains/subdomains:

http://www.tmoaradio.com runs an old CMS and also currently hosts the forum. When the forum moves to forum.tmunderground.com, it will become merely a storage site for archived radio shows, and ultimately those will go somewhere else - probably to Viewshark which also handles audio, and will allow me to put them up for streaming or download.

http://www.vee-for-vampire.com is a WordPress site, so it's low impact and can go anywhere.

http://www.tmuniversity.com currently has Owncloud running on it (put it there to test out the functionality), but once Owncloud is on its own subdomain, I'll be using tmuniversity for an install of Moodle (distance learning software) that's fairly low impact on resources.

None of the other domains are active, and when/if I do decide to use them, they will probably be nothing too extreme - perhaps WordPress or something like that. There are test WordPress installs on some of them, and you can go ahead and move them, but none are active.

Active email accounts are at tmoaradio.com, greylegion.com, and vee-for-vampire.com, and I'll need those email accounts to remain active. I don't currently use any email with tmunderground.com, though I imagine I'll have to include that once it becomes the umbrella domain.

However, I don't understand what you mean when you ask where I want to host tmunderground email accounts. If it's a virtual domain, does that mean I can't just have email accounts on that domain? And if not, what are my options? Redirect mail to/from that domain to another domain, like tmoaradio.com? Or something else. Just let me know which way I need to go on that.

I hope this has been clear enough for you guys to start the move. If anything is not clear, just let me know and I'll give you more details.

TMOAsig.png
TMU_forum01.png


#4 Ken White

Ken White

    Foul Owl on the Prowl

  • Administrators
  • 3,238 posts

Posted 16 July 2015 - 06:15 PM

The latest from Subhosting (when he mentions tmuniversity.com, he's talking about the Owncloud cloud storage deployment, not the actual TMUniversity, which will be going on server 1 with other low-impact sites:

We have set up two servers for you. Following are the server configurations:

- server1.tmunderground.com

DELL SERVER
2 x 6 CORES (12 cores total + HT)
48 GB MEMORY
4 x 480 GB SAS 15.000 RPM - RAID 10 (hardware raid)
20 TB traffic
1 Gbit network to internet

- server2.tmunderground.com

DELL SERVER
2 x 6 CORES (12 cores total + HT)
48 GB MEMORY
8 x 480 GB SAS 15.000 RPM - RAID 10 (hardware raid)
20 TB traffic
1 Gbit network to internet

Both the servers will have cPanel hosting control panel. cPanel license costs $30/mo. So it costs $60/mo for both the licenses. It will be added to your server price.

As you can see second server has twice the disk storage capacity as first server. So it would be better to host the web sites likely to have higher disk usage in server2.

At first, we will move sharemixer installation on tmunderground.com to movies.tmunderground.com in server2. After migration, we will set up a redirect from http://tmunderground.com to http://movies.tmunderground.com. You may use this redirect until you have a landing page on http://tmunderground.com with links to various subdomains.

tmoaradio.com/vb can be moved to forum.tmunderground.com in server1.

https://107.181.179.89:2199 can be moved to http://radio.tmunderground.com:2199 in server1.

tmuniversity.com will be moved to cloud.tmunderground.com in server2. How much data do you expect to have in this site for next 3-6 months?

Moodle installation on http://www.tmuniversity.com can go into server1.

Let us know if you have more information to add.

TMOAsig.png
TMU_forum01.png


#5 Ken White

Ken White

    Foul Owl on the Prowl

  • Administrators
  • 3,238 posts

Posted 17 July 2015 - 04:47 PM

More nitty-gritty, hot off the Subhosting ticket system...

We are migrating sharemixer site to movies.tmunderground.com. We will request you to stop uploading content to tmunderground.com before we switch site to movies.tmunderground.com.

movies.tmunderground.com and cloud.tmunderground.com will be hosted in server2. Otherwises will go into server1.

We should be able to put current tmoaradio.com on radio.tmunderground.com without any issues.

I was just checking with you how much data owncloud site will have in near future so that we can better plan for disk space allocation in the server. However, if you do not know the expected data usage, there is no problem; it is OK.

You can install the new movies script, maybe on beta.movies.tmunderground.com. When the new script is ready, we can rename beta.movies.tmunderground.com to movies.tmunderground.com replacing the sharemixer installation. Other than the already known tasks (database conversion, file system creation and vid file renamings), there should not be any server side issue. However, you may give us the list of server requirements for the new movies script so that we can check if hosting server can satisfy all requirements without breaking existing setup.

So when the Fear Flasher and Horror Horn appear, be ready for the first part of the move.

TMOAsig.png
TMU_forum01.png


#6 Johnny Ex

Johnny Ex

    Cliffhanger

  • Members
  • 1,736 posts
  • LocationSecret lair

Posted 17 July 2015 - 05:45 PM

I have no idea what your talking about except, move and horror slashing fire face and the word; The.
I like all of it and am on pins, not needles (they chafe me for some reason) this makes me remember the time you smashed the world and announced the inception of the SITE on the SHOW, that time....twas summer, ahh the good old NEW DAYS!
LONG LIVE THE U!
Posted Image




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users